treatmentsjavlibrary.com

ThechallengesforfutureenergysystemsDigitizationintheenergysectorcontinuesapace.
By2016,theglobalmarketforsmartgridtechnologies,whichincludessensors,managementandcontroltechnologies,communicationnetworks,andsoftware,willbeworth$80.
6billion:agrowthof28.
7%from2011.
By2020,theglobalsmartgridmarketisforecasttoexceed$400billion.
IntheEU,policiesareencouragingthedevelopmentofdecentralizedelectricitygenerationinwhichelectricExecutiveSummaryTheenergyindustryisanincreasinglydigitalindustry.
Boththeexternalmarketandinternalinfrastructurearebeingtransformedbytheemergenceofthesmartgrid.
Inthefuture,thegridwillsimplybeoneautonomousenergysystemsteeredbyanalytics:anexampleoftheInternetofThings(IoT)inaction.
Withend-to-endtransparencyofdistributionandtransmission,utilitiesandoperatorswillbebetterabletounderstandbothgridperformanceandcustomerbehavior.
ThatinsightcanbeusedtooptimizeOpExandCapExandcreatenewbusinessservices.
Thechallengewillbenotjusttogatherandsecuredatafromahugelydiverserangeofsources,butalsotomakesenseofawidevarietyofstructuredandunstructuredformats.
ThispaperconsidershowIoTtechniquesapplytoasmartgridenvironment,examinesthedatamanagement,analysis,andsecurityrequirementsandintroducestheconceptofa'datasuperstore'asthefoundationforsuccessfulgridinfrastructuresofthefuture.
vehicles,energystorageandflexibledemandareallexpectedtoplayasignificantrole.
Thisdecentralizedvision,whichenablesbi-directionalflowofelectricity,isdependentonintelligentsystemsthatdeliverbi-directionalflowofinformationtosupportpredictablefunctionsandmonitoringcapabilities.
Inaddition,newvariableslikeunexpectedandmoreextremeweatherconditions,cyber-attacksDigitizingpowerutilitiesBusinesstransformationdrivenbyadvancedanalyticsChristianDonitzkyEnergyIndustrialSolutionArchitect,IntelEMEAOliverRoosEnergyBusinessDevelopmentManager,IntelEMEAParvizPeiraviPrincipalArchitect,IntelEnterpriseSolutionSalesSylvainSautySmartGridArchitect,IntelEMEAWhitePaperGridandhighlevelsofintermittentfeed-infromrenewablespresentachallengetosystemresilience.
Theriseofprosumers,inwhichordinaryenergyconsumersalsoproduceenergyfromsmalltomid-scaleinstallations,onlyaddstothechallenge.
Makinguseofthepotentialflexibilityofboththegridanditscustomerstoovercomeconstraintsandtooptimizeperformanceof,andinvestmentin,newandexistingnetworkassetsisincreasinglyimportant.
ThesmartsecondarysubstationThedisruptioncausedbymultipleandunpredictablesourcesofrenewableenergygenerationandthedecentralizationoftheenergyinfrastructurepresentsbothchallengesandopportunitiestoutilitiesandsystemoperators.
Distributionserviceoperators(DSOs)candevelopnewbusinessmodelsandservices,butmustreorganizetheiroperationsinordertodoso.
Somehavealreadystartedonthisprocessandareexamininghowbesttodevelopandoperatetheirnetworksinthelightofthesechanges.
Thekeypointsofconsiderationforthisreorganizationare:TheneedforclosercooperationwithTransmissionServiceOperators(TSOs)toestablishgridcodesandactivelymanageandoperateasmarternetworkTheneedtobalancegenerationandconsumptionatalocallevel,whilestillplanningoperationsinconjunctionwiththoseofTSOsEnsuringinfrastructurecanbeintegratedintoEuropeanplansfortrans-nationalinterconnectionandfuturenetworkoperationTheendgoalisa'transactionalenergysystem'inwhichdecision-makingprocessestakeplaceinrealtimethankstohigh-performancedataaggregationandprocessing.
Suchatransactionalsystemrequireseffectiveworkflowmanagementandprocessesforconfiguring,switchinganddispatching,aswellasanefficientcommandandcontrolresponsesystem.
Underpinningallthisareappropriatelevelsofcyber-securityneededtoprotectcriticalinfrastructure.
Inotherwords,itneedstobe'smart'.
Thesecondarysubstationisagoodillustrationofthissmartsysteminaction.
Thetraditionalenergygridisbasedonthepremisethatpowerisgeneratedataremotepowerplantandtransmittedtowardsdomestic,commercialandindustrialconsumers.
Inthismodel,thesubstationmerelyconvertsmediumvoltagetolowvoltageanddistributesittoalimitednumberoflocalusers.
However,thearrivalofprosumersandtheirvarious,unpredictablerenewablegenerationsources,invertsthatmodelasenergyisfedbackintothegridatvariouspointsacrossit.
Inthismodel,thesecondarysubstationisamuchmorecomplexinterfacebetweentheDSO,itsconsumers,anditsprosumers.
Toperformthisnewrole,thesecondarysubstationneedstobeequippedwithsensing,communicationandcomputepoweruptoandincludingedgeanalyticsfunctions.
Thesmartgrid,dataandtheIoTThepropertiesofthesmartgridaretypicalofanInternetofThings(IoT)deployment.
AnIoTimplementationconsistsofconnecteddevices,asensornetwork,agatewayforaggregatingandTableofContentsExecutiveSummary1Thechallengesforfutureenergysystems1Thesmartsecondarysubstation.
2Thesmartgrid,dataandtheIoT.
.
.
4Fromreactivetoproactiveanalytics.
4Topologiesfordataflow.
5Securityinthecriticalinfrastructure.
5Deployingadataarchitectureforthesmartgrid5Datacollectionandmessagetransfer6Datastorage:theadventofthedatasuperstore6Eventstreamprocessing(ESP)6DistributingdatausingAPIs7Fromrawnumberstobusinessinsight.
7Conclusion.
72AnalyticsServicesConsumerAnalyticsEventAnalyticsOperationalAnalyticsFinancialandBusinessAnalyticsReportingDataAnalysisStatisticalAnalysisAppliedMachineLearningDataMiningTimeSeriesAnalysisDataVisualizationGraphAnalyticsDataArchitectureDataStagingDataDiscoveryDataModelingModelValidationDataCurationDataEngineeringDataCollectionandIntegrationDataStorageDataCleansingDataQualityDataIntegrityDataClassificationCallDataRecord(CDR)EventDataTimeSeriesDataOperationalDataMetaDataDataSourcesTransmissionLineSubstationAdvancedMetering(AMI)EngineeringThirdPartyWeather,Twittertransmittingdata,andaprivateorpubliccloud–allconnectedthroughawiredorwirelessnetwork.
Wherenewdevicesareconnected,gatewayfunctionalitycanbebuiltinsothatdataflowsremainthesame.
LikeotherIoTimplementations,thevalueofthesmartgridlieslargelyindataitproducesandtheanalysisthatitenables.
Intheexampleofthesecondarysubstation,onesubstationproducesarelativelysmalldataset:thecurrentontheprimaryandsecondaryfeeders;voltageandcurrentontheprimaryandsecondarysideofthetransformer;thetransformer'sinternaltemperature;andrealandreactivepowerindicators–whichcanhelptotracetherenewablesinjectionandmaintainrightvoltagelevelalongtheline.
However,whenthatismultipliedoverseveralhundredsubstationsitbecomesaverysubstantialdataset.
Onpaper,thearchitectureneededtoreleaseandusethisdatafromacrosstheinfrastructurelooksrelativelystraightforward.
Butoncewelookbeyondthesubstationtothewiderinfrastructure,thenumberandvarietyofdevices,frompowerplanttotransformers,transmissionanddistributionsystems,andsmartmetersatusers'premises,createanumberofspecificchallenges,namely:Designinganeffectivedatanetworkformultipledatatypes,sourcesandtreatmentsEnablingadvancedanalyticsonawidevarietyofdatasetsandsubsetsandwithindifferingtimeframesSecuringdataandcommunicationsinfrastructureinthefaceofincreasedthreatlevelsThedesignofthearchitecturealsoneedstotakeintoaccountavarietyofpotentialusecases.
Again,thesubstationisastartingpointandas'smart'capabilitiesscaletomoreFigure1:Analyticscapabilityframeworkdevicesanddifferenttypesofdevices,moredatawillbeproduced.
Figure1givessomeexamplesofwhatcanbeachievedwiththenewanalyticscapability.
Thebottomrowshowsjustsomeofthepotentialdatasourcesinthissmartenvironment:fromtransmissionlinestoexternalsourceslikeweatherreportsandevensocialmedia.
Throughtheapplicationofvariousprocesses,anumberofvalue-addedbusinessservicesaremadepossible.
Takingadvantageoftheincreasedinsightthisdataproduces,thesenewbusinessservicesandfunctionscanbebasedonconsumerbehavior,forexample,orinsightintooperationalperformance.
Operatorsandutilitieslookingtoaddsmartcapabilitiestotheirinfrastructurecanstartwiththeirchosenusecaseandthenestablishthenecessarydatasourcesanddataprocessingfunctionstodeliverit.
3Figure2showsagenericframeworkforasmarterenergysolutionandtheframeworkanalyticsthatareneededtosupportcurrentandfuturebusinesscases.
ItillustrateswhereinformationflowsfromthesubstationtotheDSOandontotheTSO,aswellastheflowbetweentheseentitiesandrenewableenergysources(RES).
Toensurethesuccessfuloperationofthissmartframework,transparencyacrosstheoperationallevelsanduptotheTSOisessential.
Inaddition,communicationandGridCodestandardsareneededtoenableseamlessdatatransmissionfromthesensortothedatamanagementsystem.
FromreactivetoproactiveanalyticsTheusecasesenabledbythesmartgriddependonricherdatasets,greateranalyticscapabilities,andnewformsofanalysis.
Whereastoday'sdatamanagementandcontrolsystemsareretrospectiveandlookatwhathashappenedandwhy,futuresystemswillallowutilitiestopredictproblemsandsotakepre-emptiveactiontoavoidthem.
Ifwegobacktotheexampleofthesecondarysubstation,themonitoringsystemscurrentlyinplacemightobserveafailureanddiagnosethataswitcherisblocked.
Inthefuture,amoresophisticatedanalyticscapabilitywouldallowtheoperatortogobeyondthisdescriptiveresponseandenableamoreproactiveandpredictivecapability.
Assystemsbecomemoreadvanced,wecouldthereforesee:PredictiveanalyticsmodelingfutureloadsothatcriticalpatternscanbeanticipatedbeforetheyhappenPrescriptiveanalyticstriggeringamaintenanceteamtoservicebeforeaminorproblembecomesacriticalsituationProactiveanalyticsenablingDSOstoenhancetheirservicetoTSOsbyprovidinginsightintoconsumerbehavior,onwhichmoreappropriatecontractsandservicescanbebasedThistransformedanalyticscapabilitywillenableoperatorstorespondtoproblemsimmediately,toplanenergydistributioninnear-realtime,andtomanagethegrid'shealthandenergygenerationinthelongerterm.
TopologiesfordataflowToensurethesepotentialbenefitsarerealized,utilitiesandsystemoperatorsneedtodesigndataflowandanalyticsappropriately.
InaccordancewithIoTdesignprinciples,therearethreemaintopologiesfordataflowandanalyticsprocessing:Cloudanalytics:inusecaseswherelatencyandresponsetimearenotcriticalfactors,adirectconnectionfromthedevicetothecloudenablesanalyticstobeperformedinthecloud.
Thisismostsuccessfulwhenlowvolumesofdataareinvolvedandthecommunicationsnetworkdoesnotbecomeoverloaded.
Someactivities,modBusmodBus/TCPIEC60870OPC-UA/FTPIEC61850PredictiveAnalytics(IntelServer,CentralAnalytics)APISCADA-SystemGridoperator-Opscenteri2SubStIntelligentSecondarySubstation(IntelHardware,IntelCorei5,EdgeAnalytics)PredictiveAnalyticsinput/outputdataPhysicalMeasurementFeeder+TransformerWeatherDataProsumerIndependentSolarGenerationI/OAggregator(IntelQuark)modBus/TCPI/OI/OI/Oi2SubStOperationalCenterofDSOSecondarysubstationOperationalDataofTSOFigure2:Thegenericframeworkforasmarterenergysolutionacrossfunctionalvoltagelevels4suchasbillingandcustomermanagement,arelikelytoremaincentrallymanagedandsoapplicationsthatbuildonthemwillalsomostlikelyneedtoberuninthecloud.
Balancedarchitecture:whenareal-timeresponsetoasimpleeventisneeded,suchasremoteactivationorshut-down,theinfrastructuredesigncallsforahybridtopologywherethesensor,actuatororgatewaycanprovidesimpleanalyticssuchasfilteringdataordetectinganomaliesinrealtime.
Thisarchitectureenablestheoperatortotakeimmediateactionatthedevicelevelandreducesdependenceontheresponsivenessandavailabilityoftheconnectiontothecloud.
Thegatewaycanalsosendbatchdatatoanintermediate'sensorcloud'forfurtheranalysis.
Severalsecondarysubstationsinthesamegeographicareacouldconnecttothesamesensorcloudtocommunicatewitheachother,forexample,withthesensorcloudsendingaggregatedatabacktothepublic/privatecloudforfurtherlonger-termanalysis.
Edgeanalytics:whenthereneedstobeareal-timeresponsetoacomplexevent,orthereisextremelylimitedbandwidthtotransferdatainrealtime,asystemthatcanperformcomplexanalyticsattheedgeispreferable.
Inthiscase,thesensor,actuator,device,orgatewayanalyzesdataautonomouslyandconnectstotheback-endcloudwheneverthetransferofbatchdataispossible.
Trendanalysisonlargeraccumulateddatasetsisperformedbytheback-endcloudandtheresultscandirectlyorindirectlychangethewaythedevicesoperate.
Thismodelisalsoappropriateforstreamliningnon-criticaldatabeforeuploadingittothecloud.
Forexample,anaggregatorofsmartmeterscouldprocessdataandsummariesofneighborhoodenergyuseina15-minuteperiod,andsendthesummarytothecloud.
Itisimportanttonotethatthereisastrongcaseforaddingmoreintelligencetosensors,actuators,anddevicestosimplifytheend-to-endinfrastructureandreducetheneedforthecostlytransferofhugeamountsofdatatothecloud.
SecurityinthecriticalinfrastructureTheneedtosecuretheinformationnetworkalongsidetheenergytransmissionnetworkisakeychallengewhenbuildingthesmartgrid,andisadeterminingfactorindatatopologydesign.
Withmachinelearningandpredictiveanalyticscomingtothefore,bothdevicesanddatarequireprotection.
So,likeanyotherexampleoftheIoT,systemdesignersneedtoensurethatallcommunicationsbetweendevicesandbetweenthegridandthecloudaresecureandcomplywithregulatoryrequirements–withoutimpedingdataflow.
Sinceexistinginfrastructurewillneedtobeprotectedalongsidethenew,includingthatwhichisnormallynot'touched'bysystemoperators,theutility'sworkflowandprocessesfortheemergingsmartgridwillneedtobedesignedtoensuretheappropriatelevelsofsecuritycanbeensured.
Thiscanbeachievedinpartthroughthedistributionofintelligenceinthesmartgrid.
Thedecentralized,bi-directionalnatureofthesmartgridmeansthatasecuritygatewaycanbeinstalledateachdataentrypoint.
Thiscanactasafirewallwhileanonymizingandencryptingsensitivedataatrestandinmotion.
However,developingatrulyend-to-endsecuritysolutionrequiresthecontributionofhardwaremanufacturersandsoftwaredevelopersandtheirabilitytocreatesecuresolutionsthatenableandprotectdataflowsandsystemintegrity.
Interoperabilitywillbeessential.
Solutionsareavailablethatenableoperatorstoimplementextendableandadaptablesecuritymeasurestoaccommodaterapidlygrowingdatavolumesandtheexpandinganalyticsenvironment.
Achievingfullsituationalawarenessacrossalldomainsofthesmartgridtodeterminewhetheranattackisinprogressisakeypriority.
TheIntelSecurityCriticalInfrastructureProtection(CIP)technologyplatformsecureslegacysystemswithinthegridaswellasnewcapabilitiesastheyareadded.
Asecuremanagedplatform,itincludesfundamentalbuildingblocksforprotectinggridinfrastructuretailoredtomachine-to-machineenvironments.
Theseincludedeviceidentity,malwareprotection,dataprotection,andresilience.
Asecurityinformationandeventmanager(SIEM)liketheIntelITSecurityBusinessIntelligenceArchitecturecanalsobeintegratedtothedatastoretobringthefullreal-timevisionandsituationalawarenessthatisrequiredtooperateasecuresmartgrid.
Itisequallyimportanttopayattentiontosecurityofthedataplatforminfrastructureandallitscomponentparts.
Forexample,whereaHadoop*5clusterisusedforstoringandprocessingdata(seepage8),componentssuchasHive*,HBase*,ClusterManagement,theHadoopfilesystem(HDFS),andfilesmustbesecured.
HereIntelAES-NIsecurityaccelerationallowsfilestobeencryptedintheHDFS(whileatrest)andsecurescommunicationsbetweennodeswithineachHadoopcluster(wheninflight).
DeployingadataarchitectureforthesmartgridHavingestablishedpotentialusecases,theanalyticsrequirements,andsecuritydemandsoftheirsmartgrid,systemoperatorsneedtodevelopanappropriatearchitecture.
Inthissection,welookatwhatsuchaninfrastructuremightlooklike.
Atadetailedfunctionallevel,thereareanumberofessentialrequirementsfortheinfrastructure,including:Theabilitytocommunicatewithavarietyofdiversedevices,plussupportformultiplecommunicationsprotocolsSupportformultipledatamodels,includingIEC61850forexchanginginformationaboutmediumandlowvoltageelectricitydistributionandtheCommonInformationModel(CIM)forexchanginginformationaboutassetsbetweenapplicationsSupportformulti-applicationandmulti-tenantenvironmentssothatdatacanbeusedfordiversebusinesspurposesCloud-baseddeliverytoensurethatsystemscanscaleondemandandwithstandfailureSupportformodularandopen-architecturephilosophy,includingtheuseofopen-sourcesolutionswhereappropriateTheabilitytocaterforgatheringandstoringdataforanalysis,aswellasexposingdatatootherapplicationsIntegrationwithexistinginfrastructureandapplicationsImportantly,datastorageandanalyticalcapabilitiesmustbeabletohandlestructured,semi-structuredandunstructureddataandcombineitwhereappropriate.
Incontrasttostructureddata,whichistypicallyVisualAnalyticApplicationODBC,JDBCTCP/IPODBCHadoopClusterSparkSparkHBaseHBaseHDFSHDFSHDFSHDFSMapreduceExternalDataSourcesWeather,SocialMedia,etc.
UtilityApplicationsGIS,CIS,MDM,DREnterpriseApplicationsSCM,CRM,ERP,BPM,etc.
InMemoryDB/AnalyticEngineODBC,JDBC,JSONEventMessagingInfrastructureEnterpriseServiceBusWiredandWirelessNetworkSmartGridInfrastructureDeviceintelligentsubstationSensorHubSensorHubMessageBrokerLoadBalancerStreamEventProcessing(SEP)NotificationCommandandControlBusinessProcessManagementBusinessInteligentAdvancedAnalyticsDatawarehouseFigure3:Proposedhybridarchitectureforanenergydatasuperstore6sourcedfromenergymanagement(EMS),distributionmanagement(DMS),ormeterdatamanagement(MDM)systems,unstructureddataincludeslessformalsourcessuchasvideoandaudiosystemsusedtoremotelymonitorthehealthandsecurityofgridassets.
Semi-structureddatafallsbetweenthetwoandcanincludedeviceconfigurationfilesinXML,amongothers.
Asthepatchworkofdatagatheredacrossthegridbecomesmorecomplex,thiswillbeadefiningfeatureofasuccessfularchitecture.
Theotherdefiningfeatureofthesmartgridisthatdataanalysisneedstobeperformedinrealtime,near-realtime,asabatchprocess,andduringstreaming.
Forexample,inaSCADAsystem,real-timedatacouldbeprocessedwithlessthanfoursecondsoflatency.
Batchprocessingcouldbeappliedtosmartmeterdatausedinbilling,whilestreaminganalysiscouldbeusedtocontinuouslymonitorthehealthandsecuritystatusofthegridinfrastructure.
Withtheseneedsinmind,thearchitectureshouldconsistofdatacollectors,aneventmessaginginfrastructure,persistentstorage,dataprocessing,appliedmachinelearninganddatamining.
ThisisshowninFigure3.
Thisinfrastructuremayalsoincludeeventstreamprocessing(ESP),advancedanalyticsusingin-memoryappliances,andanenterpriseservicebus(ESB)toenableapplicationstoexchangedatawitheachother.
Althoughthesesolutionscanenableutilitiestobuildplatformsmorequickly,integrationofdifferentcomponentsmayprovechallenging.
Apackagedsolution,basedonproprietaryoropen-sourcetechnologiesfromdifferentvendorssuchasMicrosoftAzure*andCloudera*EnterpriseDataHub,mayreducethiscomplexity.
DatacollectionandmessagetransferAswithanydistributedIoTenvironment,communicationonthesmartgridinvolvesmessagesbeingpassedbetweenvariousdevicesandnetworknodes.
Thismessage-centricapproachcantakemanyforms,fromsimpledirecttransmissionstomorecomplexmessagequeuesandtransactionalsystems.
Inallofthem,theunitofinformationexchangeisthemessageitself:theinfrastructure'sroleistoensurethatmessagesgettotheirintendedrecipients.
Amessageprocessinginfrastructureforthesmartgridshouldofferthefollowing:Cross-platforminteroperabilityDistributed,looselycoupledarchitecturethatiseasytoscaleandmanageLowlatencyandhighthroughputforpublishingandsubscribingtomessagesGuaranteedmessagedeliveryAdvancedfilteringandqueryingformessagesSupportformultiplesubscribersAutomaticloadbalancingtopreventcriticalgridconstellationsSupportforbothbatchandreal-timestreamingapplicationsMaturityandproductionreadinesswithsupport,maintenance,andcomprehensivedocumentationSupportforcommonapplicationdevelopmentenvironments(suchasScala*,Java*,andPython*)ReducednumberofserversinthedatacenterOpen-sourcebasedoptionsforanevent-messaginginfrastructure(EMI)includeKafka*,RabbitMQ*,ActiveMQ*,ZeroMQ*,JoramMQ*,HornetQ*,andDIPQ*.
Selectionagaindependsonthebusinessusecaseaswellastechnicalrequirements,forexample,theneedforsub-secondresponsetimes.
Wherethereisaneedforhigh-throughput,low-latencyconnectivitythroughwhichhundredsofmillionsofeventsaretransmittedpersecond,Kafkaisregardedastheplatformofchoice.
Itssupportforbatchandstreamingservices,andabilitytoholdanddistributelargevolumesofmessagesareimportantfeatures.
TheIntelIoTGatewayintegratestechnologiesandprotocolsfornetworking,embeddedcontrol,enterprise-gradesecurity,andeasymanageability,onwhichapplication-specificsoftwarecanrun.
Italsoenablesseamlessandsecuredataflowbetweendevicesandthecloud.
ByusingtheIntelIoTGatewaytogatherdata,operatorscantakeadvantageofpre-integrated,pre-validatedhardwareandsoftwarebuildingblockstoconnectlegacyandnewsystems.
Datastorage:theadventofthedatasuperstoreAmodernplatformabletoperformbig-dataanalyticsisanessentialcomponentofthesmartgrid.
Thedatasuperstorearchitectureprovidesaplatformforanalyticsthatenables7utilitycompaniestocollectdisparatedatasourcesandeffectivelyturnthemintobusinessinsight.
Suchaplatformcanbebuiltusingthreekeyelements:Anenterprisedatawarehouse(EDW)forinteractivequeryingofstructureddataAnApacheHadoopclusterforstoring,processingandanalyzingpoly-structureddataincludingbatch,near-realtimeandstreaminganalyticsAnin-memoryanalyticssolutiontoprovidereal-timeanalysisofdatasets,particularlythemostvaluableandsensitivesubsetsofdatastoredintheEDW.
SystemssuchasOracleExalytics*,SAPHANA*andIBMNetezza*,whichcanbebasedontheIntelXeonprocessorE7productfamily,allperformthistaskLinkingtheEDWandHadoopclustermakesitispossibletoaddressdiverseusecaserequirements.
Hadoopexcelsasahigh-speed,massivelyscalableextract,transform,andload(ETL)solution,thatcanprocesspoly-structureddata.
Theprocesseddatacanbefurtheranalyzedbynewandexistingapplications,suchasbusinessintelligence,deeplearningandmachinelearning,tosupportinteractivequeriesandotheradvancedneeds.
Withitsdistributed,parallel-processingcapabilities,theHadoopclustercanrapidlygather,storeandprocesspetabytesofpoly-structureddatabycoordinatinglocalstorageandcomputationacrosstens,hundreds,oreventhousandsofservers.
Eachserverstoresandprocessesasubsetofthedataand,becausetheapplicationsexecuteinparallel,performanceandcapacitycanscalewitheachserverthatisaddedtothecluster.
TheHadoopframeworkincludesavarietyofcomponentsformanagingdataandapplications,including:HadoopDistributedFileSystem(HDFS):afaulttolerantandself-healingdistributedfilesystemdesignedspecificallyforlarge-scaledataprocessingworkloadswherescalability,flexibility,andthroughputareessentialrequirementsMapReduce*(MR):amassivelyscalable,paralleldata-processingsoftwareframeworkthatworksintandemwithHDFSforcondensinglargevolumesofdataintousefulaggregatedresultsHBase,Cassandra*andotherNoSQLdatabases:runontopofaHadoopclusteroronaseparatecluster,thesecanextendthecapabilitiesofHadoopHive:adatawarehousesystemforHadoopthatfacilitatesdatasummarization,adhocqueries,andtheanalysisoflargedatasets,HiveprovidesamechanismforaccessingdatafromHDFSandforqueryingthedatausingaSQL-likelanguage(HiveQL)Mahout*:adata-mininglibrarythatprovidesalgorithmsforclustering,collaborativefiltering,regressiontesting,andstatisticalmodelingEventstreamprocessing(ESP)Streaminganalysisisappropriatewhenthereisacontinuousflowofdata,suchasinformationfromadvancedmeteringinfrastructure(AMI)ormeteorologicalandatmosphericreports,thatneedstobeanalyzedasitarrives.
Inadditiontocommerciallyavailablesoftware,open-sourceapplicationssuchasthein-memorySpark*andSparkStreaming*computingframeworksupporteventstreamprocessingandcanbeusedtoidentify,filter,andprocesstargetedinformation.
Theysharethesameprogramminglanguageandaframeworkthatsupports'exactlyonce'messagedeliverytoeliminatemessageloss.
SparkStreamingenablesdeveloperstowritestreamingapplicationsforthecontinuousprocessingofmicro-batchesinthesamewayaswritingbatchprocessingprogramsforSpark.
Thissimplifiesapplicationdevelopmentandgivesdatascientiststheframeworktoprovidecomprehensiveviewsbasedonreal-timeandhistoricaldata.
BothSparkandSparkStreamingleveragetheHadoopdistributedarchitectureandcanbesupportedasstandalonesolutionsorintegratedinaHadoopsolution.
DistributingdatausingAPIsProvidingaconsistentwaytoaccessandquerydataandthenexposeittoothertrustedapplicationswithoutpoint-to-pointintegration,APIsarepowerfulandflexibletoolsforintegratingandsharinginsightintobusinessprocesses.
Asaresult,theycanshortenthetimetomarketfornewsolutions,makingthemanimportantelementinthedevelopmentofdata-enabledusecasesandbusinessservices.
DemandresponseisoneexampleofhowAPIscandelivervaluethroughthesmartgrid.
Autilitycanorchestrateitsprocessing,networkandstorageresourcestoingestdifferentkindsofdata–forexamplefromsolar8photovoltaic(PV)systemsorSCADAcontrols–withdifferentlevelsoflatency.
AbrokerordispatcherwouldthentransferthedatatoanESPengineandHadoopclusterforreal-timeandbatch-orientedanalysisusingadvancedtechniques,suchasmachinelearning,forpatternandanomalydetection.
TheAPIlayerwouldthenexposetheprocesseddatatothenewgenerationofservices.
Alltheresourcesrequiredtoingest,process,analyze,anddelivertheresultingservicetobusinessuserscanbeorchestratedineitherapublicorprivatecloud.
Figure4providesahigh-levelviewoftherelevantarchitecture,fromdatasources(ontheleft)throughtobusinessservices(ontheright).
APIscanbeimplementedbyusingAPImanagementsolutionssuchasthosefromIntelMashery.
Aswiththedataitself,itisimportanttomanageandsecureAPIscentrallytoprovideflexiblebutcontrolledaccesstoinformationandresources.
OtherapplicationsthatcanbeexposedthroughanAPIinclude,butarenotlimitedto:regulatorycompliancereporting,businessintelligence,capacityplanning,consumeranalytics,andmash-upservices.
FromrawnumberstobusinessinsightAswehaveseen,thedevelopmentofthesmartgridisdrivenbyanumberofinternalandexternalfactors.
Asthispapersuggests,theadvantagesforutilitiesandoperatorsarethenewbusinessservicesthatareenabledbyinsightgainedfromadvancedanalytics.
Thedatasuperstorearchitectureprovidestheplatformforthislevelofanalyticsandenablesutilitycompaniestogatherdisparatedatasetsandturnthemintobusinessinsight.
Inanyanalyticsproject,theclaimisthat80percentofthetimeisspentondatapreparationandonly20percentisspentonmodeldevelopment,trainingandvalidation.
Thebigdatatechnologiesoutlinedherenotonlyprovidethescalabledatastorageandprocessingcapabilities,theygivedatascientistsdirectaccesstoentiredatasets–andsoacceleratedataanalysis.
Asaresult,datascientistscanrunconcurrentanalysisandsimulationswithamuchshortertimetocompletion.
Analyticsservicesthathavenotbeenviableuntilnow,suchasreal-timedetectionofanomaliesandcustomerbehavioralanalysis,arenowpossible.
Byincorporatingreadilyavailabledatafromexternalsources,utilitiesareabletoaddanotherlayerofinsightandpushfurtherintopredictiveandprescriptiveanalytics.
Forexample,theycanmanageenergyprocurementAnalyticCloudPlatformStreamEventProcessingCEPEngineEnterpriseDataHubHadoopClusterEnterpriseServiceBusDWODSEnterpriseApplicationsDataSourcesTSODSOintelligent2ndsubstationSmartMeterAMIElectricalVehiclesLegacyFieldSystemsPartnerDataExternalDataSourcesSocialMedia,etc.
ExternalCloud(PublicorPrivate)DataSubscriberDataSubscriberServicesDataasaserviceConsumerAnalyticsOperationalOpenDataDashboardforBIExternalAnalyticsRegulatoryComplianceDemandResponseProgramCapacityPlanning,LoadForecastLoadBalancerAPIExposureFigure4:Schemaofanenergydatasuperstore9withgreaterprecision,basedonanunderstandingofdemandandpricesorpredictpotentialoutagesandequipmentfailuresandtakeimmediatepreventativeaction.
Havingabetterinsightintohowmuchenergywillberequiredinaparticularlocationenablesutilitiestomoreeffectivelyplanforgenerating,buying,ordistributingelectricitytothatlocation.
Havinganunderstandingofenergyconsumptionandrenewableenergyinjectionatthesubstationleveloffersalevelofinsightsimilartothatfromsmartmetersinhundredsofhomes,butatlowercost.
ConclusionItisalmostimpossibletoexaggeratethetransformationaleffectofthesmartgridonenergygeneration.
Itreleasesvaluabledatafromeverypointofthephysicalinfrastructure,andprovidesthemechanismswherebythatdatacanbeconvertedintoextraordinaryinsightandunderstandingintoeveryaspectofthebusiness.
Utilitiesandsystemoperatorshavethepotentialtobecomedatapowerhouses:processinggigabytesofdataasgigawattsofpowertraversethenetwork.
Butifthispotentialistoberealized,buildingthenewdata-drivenoperationmuststartnow.
Asthispaperhasdemonstrated,thevolume,varietyandvelocityofdatainvolvedpresentssignificantchallenges:notjusttothosewhomustdesignthereferencearchitectureforhandlingandanalyzingdata,butthoseinchargeofprotectingandsecuringitinthefaceofincreasedthreatlevels.
Thedatasuperstorepresentedhererepresentsakeybuildingblockforthenewsmartgrid–andoneforwhichthetechnologyandcapabilitytobuildisalreadyavailable.
Enablingdatatobecapturedandanalyzed,queriedinrealtimeifnecessary,andcombinedflexiblytodeliveruniquenewinsights,itremovestheneedtodevelopdifferentarchitecturesforthevariousdifferentdatatypes.
Basedonopensourcecomponentswhereavailableitbuildsinsecurityandinteroperabilityateverylayer.
Withthedatasuperstoreinplace,utilitieswillbeabletodevelopnewinformation-driven,value-addedbusinessservices,aswellasdeployingthepredictive,proactiveandpreventiveanalyticsthatwilldrivetechnical,operationalandenergyefficienciesthroughoutthegrid.
www.
marketsandmarkets.
com/Market-Reports/smart-grid-technology-application-market-453.
htmlgclid=CP660aHV7L4CFaXHtAod-1MA4Awww.
greentechmedia.
com/articles/read/smart-grid-market-to-surpass-400-billion-worldwide-by-2020www.
mcafee.
com/ca/about/news/2015/q1/20150304-01.
aspxwww.
intel.
com/content/www/us/en/it-management/intel-it-best-practices/security-business-intelligence-siem-video.
htmlThesesecurityfeaturesarebasedontheopensourceprojectRhino,whichisavailablefrommanagedopensourceHadoopvendorssuchasClouderaInteltechnologies'featuresandbenefitsdependonsystemconfigurationandmayrequireenabledhardware,softwareorserviceactivation.
Performancevariesdependingonsystemconfiguration.
Nocomputersystemcanbeabsolutelysecure.
Checkwithyoursystemmanufacturerorretailerorlearnmoreatwww.
intel.
comIntel,theIntellogo,IntelXeon,IntelIoTGateway,IntelMashery,andIntelAES-NIaretrademarksofIntelCorporationintheU.
S.
andothercountries.
*Othernamesandbrandsmaybeclaimedasthepropertyofothers.
2015,IntelCorporationPleaseRecycle