ThechallengesforfutureenergysystemsDigitizationintheenergysectorcontinuesapace.
By2016,theglobalmarketforsmartgridtechnologies,whichincludessensors,managementandcontroltechnologies,communicationnetworks,andsoftware,willbeworth$80.
6billion:agrowthof28.
7%from2011.
By2020,theglobalsmartgridmarketisforecasttoexceed$400billion.
IntheEU,policiesareencouragingthedevelopmentofdecentralizedelectricitygenerationinwhichelectricExecutiveSummaryTheenergyindustryisanincreasinglydigitalindustry.
Boththeexternalmarketandinternalinfrastructurearebeingtransformedbytheemergenceofthesmartgrid.
Inthefuture,thegridwillsimplybeoneautonomousenergysystemsteeredbyanalytics:anexampleoftheInternetofThings(IoT)inaction.
Withend-to-endtransparencyofdistributionandtransmission,utilitiesandoperatorswillbebetterabletounderstandbothgridperformanceandcustomerbehavior.
ThatinsightcanbeusedtooptimizeOpExandCapExandcreatenewbusinessservices.
Thechallengewillbenotjusttogatherandsecuredatafromahugelydiverserangeofsources,butalsotomakesenseofawidevarietyofstructuredandunstructuredformats.
ThispaperconsidershowIoTtechniquesapplytoasmartgridenvironment,examinesthedatamanagement,analysis,andsecurityrequirementsandintroducestheconceptofa'datasuperstore'asthefoundationforsuccessfulgridinfrastructuresofthefuture.
vehicles,energystorageandflexibledemandareallexpectedtoplayasignificantrole.
Thisdecentralizedvision,whichenablesbi-directionalflowofelectricity,isdependentonintelligentsystemsthatdeliverbi-directionalflowofinformationtosupportpredictablefunctionsandmonitoringcapabilities.
Inaddition,newvariableslikeunexpectedandmoreextremeweatherconditions,cyber-attacksDigitizingpowerutilitiesBusinesstransformationdrivenbyadvancedanalyticsChristianDonitzkyEnergyIndustrialSolutionArchitect,IntelEMEAOliverRoosEnergyBusinessDevelopmentManager,IntelEMEAParvizPeiraviPrincipalArchitect,IntelEnterpriseSolutionSalesSylvainSautySmartGridArchitect,IntelEMEAWhitePaperGridandhighlevelsofintermittentfeed-infromrenewablespresentachallengetosystemresilience.
Theriseofprosumers,inwhichordinaryenergyconsumersalsoproduceenergyfromsmalltomid-scaleinstallations,onlyaddstothechallenge.
Makinguseofthepotentialflexibilityofboththegridanditscustomerstoovercomeconstraintsandtooptimizeperformanceof,andinvestmentin,newandexistingnetworkassetsisincreasinglyimportant.
ThesmartsecondarysubstationThedisruptioncausedbymultipleandunpredictablesourcesofrenewableenergygenerationandthedecentralizationoftheenergyinfrastructurepresentsbothchallengesandopportunitiestoutilitiesandsystemoperators.
Distributionserviceoperators(DSOs)candevelopnewbusinessmodelsandservices,butmustreorganizetheiroperationsinordertodoso.
Somehavealreadystartedonthisprocessandareexamininghowbesttodevelopandoperatetheirnetworksinthelightofthesechanges.
Thekeypointsofconsiderationforthisreorganizationare:TheneedforclosercooperationwithTransmissionServiceOperators(TSOs)toestablishgridcodesandactivelymanageandoperateasmarternetworkTheneedtobalancegenerationandconsumptionatalocallevel,whilestillplanningoperationsinconjunctionwiththoseofTSOsEnsuringinfrastructurecanbeintegratedintoEuropeanplansfortrans-nationalinterconnectionandfuturenetworkoperationTheendgoalisa'transactionalenergysystem'inwhichdecision-makingprocessestakeplaceinrealtimethankstohigh-performancedataaggregationandprocessing.
Suchatransactionalsystemrequireseffectiveworkflowmanagementandprocessesforconfiguring,switchinganddispatching,aswellasanefficientcommandandcontrolresponsesystem.
Underpinningallthisareappropriatelevelsofcyber-securityneededtoprotectcriticalinfrastructure.
Inotherwords,itneedstobe'smart'.
Thesecondarysubstationisagoodillustrationofthissmartsysteminaction.
Thetraditionalenergygridisbasedonthepremisethatpowerisgeneratedataremotepowerplantandtransmittedtowardsdomestic,commercialandindustrialconsumers.
Inthismodel,thesubstationmerelyconvertsmediumvoltagetolowvoltageanddistributesittoalimitednumberoflocalusers.
However,thearrivalofprosumersandtheirvarious,unpredictablerenewablegenerationsources,invertsthatmodelasenergyisfedbackintothegridatvariouspointsacrossit.
Inthismodel,thesecondarysubstationisamuchmorecomplexinterfacebetweentheDSO,itsconsumers,anditsprosumers.
Toperformthisnewrole,thesecondarysubstationneedstobeequippedwithsensing,communicationandcomputepoweruptoandincludingedgeanalyticsfunctions.
Thesmartgrid,dataandtheIoTThepropertiesofthesmartgridaretypicalofanInternetofThings(IoT)deployment.
AnIoTimplementationconsistsofconnecteddevices,asensornetwork,agatewayforaggregatingandTableofContentsExecutiveSummary1Thechallengesforfutureenergysystems1Thesmartsecondarysubstation.
2Thesmartgrid,dataandtheIoT.
.
.
4Fromreactivetoproactiveanalytics.
4Topologiesfordataflow.
5Securityinthecriticalinfrastructure.
5Deployingadataarchitectureforthesmartgrid5Datacollectionandmessagetransfer6Datastorage:theadventofthedatasuperstore6Eventstreamprocessing(ESP)6DistributingdatausingAPIs7Fromrawnumberstobusinessinsight.
7Conclusion.
72AnalyticsServicesConsumerAnalyticsEventAnalyticsOperationalAnalyticsFinancialandBusinessAnalyticsReportingDataAnalysisStatisticalAnalysisAppliedMachineLearningDataMiningTimeSeriesAnalysisDataVisualizationGraphAnalyticsDataArchitectureDataStagingDataDiscoveryDataModelingModelValidationDataCurationDataEngineeringDataCollectionandIntegrationDataStorageDataCleansingDataQualityDataIntegrityDataClassificationCallDataRecord(CDR)EventDataTimeSeriesDataOperationalDataMetaDataDataSourcesTransmissionLineSubstationAdvancedMetering(AMI)EngineeringThirdPartyWeather,Twittertransmittingdata,andaprivateorpubliccloud–allconnectedthroughawiredorwirelessnetwork.
Wherenewdevicesareconnected,gatewayfunctionalitycanbebuiltinsothatdataflowsremainthesame.
LikeotherIoTimplementations,thevalueofthesmartgridlieslargelyindataitproducesandtheanalysisthatitenables.
Intheexampleofthesecondarysubstation,onesubstationproducesarelativelysmalldataset:thecurrentontheprimaryandsecondaryfeeders;voltageandcurrentontheprimaryandsecondarysideofthetransformer;thetransformer'sinternaltemperature;andrealandreactivepowerindicators–whichcanhelptotracetherenewablesinjectionandmaintainrightvoltagelevelalongtheline.
However,whenthatismultipliedoverseveralhundredsubstationsitbecomesaverysubstantialdataset.
Onpaper,thearchitectureneededtoreleaseandusethisdatafromacrosstheinfrastructurelooksrelativelystraightforward.
Butoncewelookbeyondthesubstationtothewiderinfrastructure,thenumberandvarietyofdevices,frompowerplanttotransformers,transmissionanddistributionsystems,andsmartmetersatusers'premises,createanumberofspecificchallenges,namely:Designinganeffectivedatanetworkformultipledatatypes,sourcesandtreatmentsEnablingadvancedanalyticsonawidevarietyofdatasetsandsubsetsandwithindifferingtimeframesSecuringdataandcommunicationsinfrastructureinthefaceofincreasedthreatlevelsThedesignofthearchitecturealsoneedstotakeintoaccountavarietyofpotentialusecases.
Again,thesubstationisastartingpointandas'smart'capabilitiesscaletomoreFigure1:Analyticscapabilityframeworkdevicesanddifferenttypesofdevices,moredatawillbeproduced.
Figure1givessomeexamplesofwhatcanbeachievedwiththenewanalyticscapability.
Thebottomrowshowsjustsomeofthepotentialdatasourcesinthissmartenvironment:fromtransmissionlinestoexternalsourceslikeweatherreportsandevensocialmedia.
Throughtheapplicationofvariousprocesses,anumberofvalue-addedbusinessservicesaremadepossible.
Takingadvantageoftheincreasedinsightthisdataproduces,thesenewbusinessservicesandfunctionscanbebasedonconsumerbehavior,forexample,orinsightintooperationalperformance.
Operatorsandutilitieslookingtoaddsmartcapabilitiestotheirinfrastructurecanstartwiththeirchosenusecaseandthenestablishthenecessarydatasourcesanddataprocessingfunctionstodeliverit.
3Figure2showsagenericframeworkforasmarterenergysolutionandtheframeworkanalyticsthatareneededtosupportcurrentandfuturebusinesscases.
ItillustrateswhereinformationflowsfromthesubstationtotheDSOandontotheTSO,aswellastheflowbetweentheseentitiesandrenewableenergysources(RES).
Toensurethesuccessfuloperationofthissmartframework,transparencyacrosstheoperationallevelsanduptotheTSOisessential.
Inaddition,communicationandGridCodestandardsareneededtoenableseamlessdatatransmissionfromthesensortothedatamanagementsystem.
FromreactivetoproactiveanalyticsTheusecasesenabledbythesmartgriddependonricherdatasets,greateranalyticscapabilities,andnewformsofanalysis.
Whereastoday'sdatamanagementandcontrolsystemsareretrospectiveandlookatwhathashappenedandwhy,futuresystemswillallowutilitiestopredictproblemsandsotakepre-emptiveactiontoavoidthem.
Ifwegobacktotheexampleofthesecondarysubstation,themonitoringsystemscurrentlyinplacemightobserveafailureanddiagnosethataswitcherisblocked.
Inthefuture,amoresophisticatedanalyticscapabilitywouldallowtheoperatortogobeyondthisdescriptiveresponseandenableamoreproactiveandpredictivecapability.
Assystemsbecomemoreadvanced,wecouldthereforesee:PredictiveanalyticsmodelingfutureloadsothatcriticalpatternscanbeanticipatedbeforetheyhappenPrescriptiveanalyticstriggeringamaintenanceteamtoservicebeforeaminorproblembecomesacriticalsituationProactiveanalyticsenablingDSOstoenhancetheirservicetoTSOsbyprovidinginsightintoconsumerbehavior,onwhichmoreappropriatecontractsandservicescanbebasedThistransformedanalyticscapabilitywillenableoperatorstorespondtoproblemsimmediately,toplanenergydistributioninnear-realtime,andtomanagethegrid'shealthandenergygenerationinthelongerterm.
TopologiesfordataflowToensurethesepotentialbenefitsarerealized,utilitiesandsystemoperatorsneedtodesigndataflowandanalyticsappropriately.
InaccordancewithIoTdesignprinciples,therearethreemaintopologiesfordataflowandanalyticsprocessing:Cloudanalytics:inusecaseswherelatencyandresponsetimearenotcriticalfactors,adirectconnectionfromthedevicetothecloudenablesanalyticstobeperformedinthecloud.
Thisismostsuccessfulwhenlowvolumesofdataareinvolvedandthecommunicationsnetworkdoesnotbecomeoverloaded.
Someactivities,modBusmodBus/TCPIEC60870OPC-UA/FTPIEC61850PredictiveAnalytics(IntelServer,CentralAnalytics)APISCADA-SystemGridoperator-Opscenteri2SubStIntelligentSecondarySubstation(IntelHardware,IntelCorei5,EdgeAnalytics)PredictiveAnalyticsinput/outputdataPhysicalMeasurementFeeder+TransformerWeatherDataProsumerIndependentSolarGenerationI/OAggregator(IntelQuark)modBus/TCPI/OI/OI/Oi2SubStOperationalCenterofDSOSecondarysubstationOperationalDataofTSOFigure2:Thegenericframeworkforasmarterenergysolutionacrossfunctionalvoltagelevels4suchasbillingandcustomermanagement,arelikelytoremaincentrallymanagedandsoapplicationsthatbuildonthemwillalsomostlikelyneedtoberuninthecloud.
Balancedarchitecture:whenareal-timeresponsetoasimpleeventisneeded,suchasremoteactivationorshut-down,theinfrastructuredesigncallsforahybridtopologywherethesensor,actuatororgatewaycanprovidesimpleanalyticssuchasfilteringdataordetectinganomaliesinrealtime.
Thisarchitectureenablestheoperatortotakeimmediateactionatthedevicelevelandreducesdependenceontheresponsivenessandavailabilityoftheconnectiontothecloud.
Thegatewaycanalsosendbatchdatatoanintermediate'sensorcloud'forfurtheranalysis.
Severalsecondarysubstationsinthesamegeographicareacouldconnecttothesamesensorcloudtocommunicatewitheachother,forexample,withthesensorcloudsendingaggregatedatabacktothepublic/privatecloudforfurtherlonger-termanalysis.
Edgeanalytics:whenthereneedstobeareal-timeresponsetoacomplexevent,orthereisextremelylimitedbandwidthtotransferdatainrealtime,asystemthatcanperformcomplexanalyticsattheedgeispreferable.
Inthiscase,thesensor,actuator,device,orgatewayanalyzesdataautonomouslyandconnectstotheback-endcloudwheneverthetransferofbatchdataispossible.
Trendanalysisonlargeraccumulateddatasetsisperformedbytheback-endcloudandtheresultscandirectlyorindirectlychangethewaythedevicesoperate.
Thismodelisalsoappropriateforstreamliningnon-criticaldatabeforeuploadingittothecloud.
Forexample,anaggregatorofsmartmeterscouldprocessdataandsummariesofneighborhoodenergyuseina15-minuteperiod,andsendthesummarytothecloud.
Itisimportanttonotethatthereisastrongcaseforaddingmoreintelligencetosensors,actuators,anddevicestosimplifytheend-to-endinfrastructureandreducetheneedforthecostlytransferofhugeamountsofdatatothecloud.
SecurityinthecriticalinfrastructureTheneedtosecuretheinformationnetworkalongsidetheenergytransmissionnetworkisakeychallengewhenbuildingthesmartgrid,andisadeterminingfactorindatatopologydesign.
Withmachinelearningandpredictiveanalyticscomingtothefore,bothdevicesanddatarequireprotection.
So,likeanyotherexampleoftheIoT,systemdesignersneedtoensurethatallcommunicationsbetweendevicesandbetweenthegridandthecloudaresecureandcomplywithregulatoryrequirements–withoutimpedingdataflow.
Sinceexistinginfrastructurewillneedtobeprotectedalongsidethenew,includingthatwhichisnormallynot'touched'bysystemoperators,theutility'sworkflowandprocessesfortheemergingsmartgridwillneedtobedesignedtoensuretheappropriatelevelsofsecuritycanbeensured.
Thiscanbeachievedinpartthroughthedistributionofintelligenceinthesmartgrid.
Thedecentralized,bi-directionalnatureofthesmartgridmeansthatasecuritygatewaycanbeinstalledateachdataentrypoint.
Thiscanactasafirewallwhileanonymizingandencryptingsensitivedataatrestandinmotion.
However,developingatrulyend-to-endsecuritysolutionrequiresthecontributionofhardwaremanufacturersandsoftwaredevelopersandtheirabilitytocreatesecuresolutionsthatenableandprotectdataflowsandsystemintegrity.
Interoperabilitywillbeessential.
Solutionsareavailablethatenableoperatorstoimplementextendableandadaptablesecuritymeasurestoaccommodaterapidlygrowingdatavolumesandtheexpandinganalyticsenvironment.
Achievingfullsituationalawarenessacrossalldomainsofthesmartgridtodeterminewhetheranattackisinprogressisakeypriority.
TheIntelSecurityCriticalInfrastructureProtection(CIP)technologyplatformsecureslegacysystemswithinthegridaswellasnewcapabilitiesastheyareadded.
Asecuremanagedplatform,itincludesfundamentalbuildingblocksforprotectinggridinfrastructuretailoredtomachine-to-machineenvironments.
Theseincludedeviceidentity,malwareprotection,dataprotection,andresilience.
Asecurityinformationandeventmanager(SIEM)liketheIntelITSecurityBusinessIntelligenceArchitecturecanalsobeintegratedtothedatastoretobringthefullreal-timevisionandsituationalawarenessthatisrequiredtooperateasecuresmartgrid.
Itisequallyimportanttopayattentiontosecurityofthedataplatforminfrastructureandallitscomponentparts.
Forexample,whereaHadoop*5clusterisusedforstoringandprocessingdata(seepage8),componentssuchasHive*,HBase*,ClusterManagement,theHadoopfilesystem(HDFS),andfilesmustbesecured.
HereIntelAES-NIsecurityaccelerationallowsfilestobeencryptedintheHDFS(whileatrest)andsecurescommunicationsbetweennodeswithineachHadoopcluster(wheninflight).
DeployingadataarchitectureforthesmartgridHavingestablishedpotentialusecases,theanalyticsrequirements,andsecuritydemandsoftheirsmartgrid,systemoperatorsneedtodevelopanappropriatearchitecture.
Inthissection,welookatwhatsuchaninfrastructuremightlooklike.
Atadetailedfunctionallevel,thereareanumberofessentialrequirementsfortheinfrastructure,including:Theabilitytocommunicatewithavarietyofdiversedevices,plussupportformultiplecommunicationsprotocolsSupportformultipledatamodels,includingIEC61850forexchanginginformationaboutmediumandlowvoltageelectricitydistributionandtheCommonInformationModel(CIM)forexchanginginformationaboutassetsbetweenapplicationsSupportformulti-applicationandmulti-tenantenvironmentssothatdatacanbeusedfordiversebusinesspurposesCloud-baseddeliverytoensurethatsystemscanscaleondemandandwithstandfailureSupportformodularandopen-architecturephilosophy,includingtheuseofopen-sourcesolutionswhereappropriateTheabilitytocaterforgatheringandstoringdataforanalysis,aswellasexposingdatatootherapplicationsIntegrationwithexistinginfrastructureandapplicationsImportantly,datastorageandanalyticalcapabilitiesmustbeabletohandlestructured,semi-structuredandunstructureddataandcombineitwhereappropriate.
Incontrasttostructureddata,whichistypicallyVisualAnalyticApplicationODBC,JDBCTCP/IPODBCHadoopClusterSparkSparkHBaseHBaseHDFSHDFSHDFSHDFSMapreduceExternalDataSourcesWeather,SocialMedia,etc.
UtilityApplicationsGIS,CIS,MDM,DREnterpriseApplicationsSCM,CRM,ERP,BPM,etc.
InMemoryDB/AnalyticEngineODBC,JDBC,JSONEventMessagingInfrastructureEnterpriseServiceBusWiredandWirelessNetworkSmartGridInfrastructureDeviceintelligentsubstationSensorHubSensorHubMessageBrokerLoadBalancerStreamEventProcessing(SEP)NotificationCommandandControlBusinessProcessManagementBusinessInteligentAdvancedAnalyticsDatawarehouseFigure3:Proposedhybridarchitectureforanenergydatasuperstore6sourcedfromenergymanagement(EMS),distributionmanagement(DMS),ormeterdatamanagement(MDM)systems,unstructureddataincludeslessformalsourcessuchasvideoandaudiosystemsusedtoremotelymonitorthehealthandsecurityofgridassets.
Semi-structureddatafallsbetweenthetwoandcanincludedeviceconfigurationfilesinXML,amongothers.
Asthepatchworkofdatagatheredacrossthegridbecomesmorecomplex,thiswillbeadefiningfeatureofasuccessfularchitecture.
Theotherdefiningfeatureofthesmartgridisthatdataanalysisneedstobeperformedinrealtime,near-realtime,asabatchprocess,andduringstreaming.
Forexample,inaSCADAsystem,real-timedatacouldbeprocessedwithlessthanfoursecondsoflatency.
Batchprocessingcouldbeappliedtosmartmeterdatausedinbilling,whilestreaminganalysiscouldbeusedtocontinuouslymonitorthehealthandsecuritystatusofthegridinfrastructure.
Withtheseneedsinmind,thearchitectureshouldconsistofdatacollectors,aneventmessaginginfrastructure,persistentstorage,dataprocessing,appliedmachinelearninganddatamining.
ThisisshowninFigure3.
Thisinfrastructuremayalsoincludeeventstreamprocessing(ESP),advancedanalyticsusingin-memoryappliances,andanenterpriseservicebus(ESB)toenableapplicationstoexchangedatawitheachother.
Althoughthesesolutionscanenableutilitiestobuildplatformsmorequickly,integrationofdifferentcomponentsmayprovechallenging.
Apackagedsolution,basedonproprietaryoropen-sourcetechnologiesfromdifferentvendorssuchasMicrosoftAzure*andCloudera*EnterpriseDataHub,mayreducethiscomplexity.
DatacollectionandmessagetransferAswithanydistributedIoTenvironment,communicationonthesmartgridinvolvesmessagesbeingpassedbetweenvariousdevicesandnetworknodes.
Thismessage-centricapproachcantakemanyforms,fromsimpledirecttransmissionstomorecomplexmessagequeuesandtransactionalsystems.
Inallofthem,theunitofinformationexchangeisthemessageitself:theinfrastructure'sroleistoensurethatmessagesgettotheirintendedrecipients.
Amessageprocessinginfrastructureforthesmartgridshouldofferthefollowing:Cross-platforminteroperabilityDistributed,looselycoupledarchitecturethatiseasytoscaleandmanageLowlatencyandhighthroughputforpublishingandsubscribingtomessagesGuaranteedmessagedeliveryAdvancedfilteringandqueryingformessagesSupportformultiplesubscribersAutomaticloadbalancingtopreventcriticalgridconstellationsSupportforbothbatchandreal-timestreamingapplicationsMaturityandproductionreadinesswithsupport,maintenance,andcomprehensivedocumentationSupportforcommonapplicationdevelopmentenvironments(suchasScala*,Java*,andPython*)ReducednumberofserversinthedatacenterOpen-sourcebasedoptionsforanevent-messaginginfrastructure(EMI)includeKafka*,RabbitMQ*,ActiveMQ*,ZeroMQ*,JoramMQ*,HornetQ*,andDIPQ*.
Selectionagaindependsonthebusinessusecaseaswellastechnicalrequirements,forexample,theneedforsub-secondresponsetimes.
Wherethereisaneedforhigh-throughput,low-latencyconnectivitythroughwhichhundredsofmillionsofeventsaretransmittedpersecond,Kafkaisregardedastheplatformofchoice.
Itssupportforbatchandstreamingservices,andabilitytoholdanddistributelargevolumesofmessagesareimportantfeatures.
TheIntelIoTGatewayintegratestechnologiesandprotocolsfornetworking,embeddedcontrol,enterprise-gradesecurity,andeasymanageability,onwhichapplication-specificsoftwarecanrun.
Italsoenablesseamlessandsecuredataflowbetweendevicesandthecloud.
ByusingtheIntelIoTGatewaytogatherdata,operatorscantakeadvantageofpre-integrated,pre-validatedhardwareandsoftwarebuildingblockstoconnectlegacyandnewsystems.
Datastorage:theadventofthedatasuperstoreAmodernplatformabletoperformbig-dataanalyticsisanessentialcomponentofthesmartgrid.
Thedatasuperstorearchitectureprovidesaplatformforanalyticsthatenables7utilitycompaniestocollectdisparatedatasourcesandeffectivelyturnthemintobusinessinsight.
Suchaplatformcanbebuiltusingthreekeyelements:Anenterprisedatawarehouse(EDW)forinteractivequeryingofstructureddataAnApacheHadoopclusterforstoring,processingandanalyzingpoly-structureddataincludingbatch,near-realtimeandstreaminganalyticsAnin-memoryanalyticssolutiontoprovidereal-timeanalysisofdatasets,particularlythemostvaluableandsensitivesubsetsofdatastoredintheEDW.
SystemssuchasOracleExalytics*,SAPHANA*andIBMNetezza*,whichcanbebasedontheIntelXeonprocessorE7productfamily,allperformthistaskLinkingtheEDWandHadoopclustermakesitispossibletoaddressdiverseusecaserequirements.
Hadoopexcelsasahigh-speed,massivelyscalableextract,transform,andload(ETL)solution,thatcanprocesspoly-structureddata.
Theprocesseddatacanbefurtheranalyzedbynewandexistingapplications,suchasbusinessintelligence,deeplearningandmachinelearning,tosupportinteractivequeriesandotheradvancedneeds.
Withitsdistributed,parallel-processingcapabilities,theHadoopclustercanrapidlygather,storeandprocesspetabytesofpoly-structureddatabycoordinatinglocalstorageandcomputationacrosstens,hundreds,oreventhousandsofservers.
Eachserverstoresandprocessesasubsetofthedataand,becausetheapplicationsexecuteinparallel,performanceandcapacitycanscalewitheachserverthatisaddedtothecluster.
TheHadoopframeworkincludesavarietyofcomponentsformanagingdataandapplications,including:HadoopDistributedFileSystem(HDFS):afaulttolerantandself-healingdistributedfilesystemdesignedspecificallyforlarge-scaledataprocessingworkloadswherescalability,flexibility,andthroughputareessentialrequirementsMapReduce*(MR):amassivelyscalable,paralleldata-processingsoftwareframeworkthatworksintandemwithHDFSforcondensinglargevolumesofdataintousefulaggregatedresultsHBase,Cassandra*andotherNoSQLdatabases:runontopofaHadoopclusteroronaseparatecluster,thesecanextendthecapabilitiesofHadoopHive:adatawarehousesystemforHadoopthatfacilitatesdatasummarization,adhocqueries,andtheanalysisoflargedatasets,HiveprovidesamechanismforaccessingdatafromHDFSandforqueryingthedatausingaSQL-likelanguage(HiveQL)Mahout*:adata-mininglibrarythatprovidesalgorithmsforclustering,collaborativefiltering,regressiontesting,andstatisticalmodelingEventstreamprocessing(ESP)Streaminganalysisisappropriatewhenthereisacontinuousflowofdata,suchasinformationfromadvancedmeteringinfrastructure(AMI)ormeteorologicalandatmosphericreports,thatneedstobeanalyzedasitarrives.
Inadditiontocommerciallyavailablesoftware,open-sourceapplicationssuchasthein-memorySpark*andSparkStreaming*computingframeworksupporteventstreamprocessingandcanbeusedtoidentify,filter,andprocesstargetedinformation.
Theysharethesameprogramminglanguageandaframeworkthatsupports'exactlyonce'messagedeliverytoeliminatemessageloss.
SparkStreamingenablesdeveloperstowritestreamingapplicationsforthecontinuousprocessingofmicro-batchesinthesamewayaswritingbatchprocessingprogramsforSpark.
Thissimplifiesapplicationdevelopmentandgivesdatascientiststheframeworktoprovidecomprehensiveviewsbasedonreal-timeandhistoricaldata.
BothSparkandSparkStreamingleveragetheHadoopdistributedarchitectureandcanbesupportedasstandalonesolutionsorintegratedinaHadoopsolution.
DistributingdatausingAPIsProvidingaconsistentwaytoaccessandquerydataandthenexposeittoothertrustedapplicationswithoutpoint-to-pointintegration,APIsarepowerfulandflexibletoolsforintegratingandsharinginsightintobusinessprocesses.
Asaresult,theycanshortenthetimetomarketfornewsolutions,makingthemanimportantelementinthedevelopmentofdata-enabledusecasesandbusinessservices.
DemandresponseisoneexampleofhowAPIscandelivervaluethroughthesmartgrid.
Autilitycanorchestrateitsprocessing,networkandstorageresourcestoingestdifferentkindsofdata–forexamplefromsolar8photovoltaic(PV)systemsorSCADAcontrols–withdifferentlevelsoflatency.
AbrokerordispatcherwouldthentransferthedatatoanESPengineandHadoopclusterforreal-timeandbatch-orientedanalysisusingadvancedtechniques,suchasmachinelearning,forpatternandanomalydetection.
TheAPIlayerwouldthenexposetheprocesseddatatothenewgenerationofservices.
Alltheresourcesrequiredtoingest,process,analyze,anddelivertheresultingservicetobusinessuserscanbeorchestratedineitherapublicorprivatecloud.
Figure4providesahigh-levelviewoftherelevantarchitecture,fromdatasources(ontheleft)throughtobusinessservices(ontheright).
APIscanbeimplementedbyusingAPImanagementsolutionssuchasthosefromIntelMashery.
Aswiththedataitself,itisimportanttomanageandsecureAPIscentrallytoprovideflexiblebutcontrolledaccesstoinformationandresources.
OtherapplicationsthatcanbeexposedthroughanAPIinclude,butarenotlimitedto:regulatorycompliancereporting,businessintelligence,capacityplanning,consumeranalytics,andmash-upservices.
FromrawnumberstobusinessinsightAswehaveseen,thedevelopmentofthesmartgridisdrivenbyanumberofinternalandexternalfactors.
Asthispapersuggests,theadvantagesforutilitiesandoperatorsarethenewbusinessservicesthatareenabledbyinsightgainedfromadvancedanalytics.
Thedatasuperstorearchitectureprovidestheplatformforthislevelofanalyticsandenablesutilitycompaniestogatherdisparatedatasetsandturnthemintobusinessinsight.
Inanyanalyticsproject,theclaimisthat80percentofthetimeisspentondatapreparationandonly20percentisspentonmodeldevelopment,trainingandvalidation.
Thebigdatatechnologiesoutlinedherenotonlyprovidethescalabledatastorageandprocessingcapabilities,theygivedatascientistsdirectaccesstoentiredatasets–andsoacceleratedataanalysis.
Asaresult,datascientistscanrunconcurrentanalysisandsimulationswithamuchshortertimetocompletion.
Analyticsservicesthathavenotbeenviableuntilnow,suchasreal-timedetectionofanomaliesandcustomerbehavioralanalysis,arenowpossible.
Byincorporatingreadilyavailabledatafromexternalsources,utilitiesareabletoaddanotherlayerofinsightandpushfurtherintopredictiveandprescriptiveanalytics.
Forexample,theycanmanageenergyprocurementAnalyticCloudPlatformStreamEventProcessingCEPEngineEnterpriseDataHubHadoopClusterEnterpriseServiceBusDWODSEnterpriseApplicationsDataSourcesTSODSOintelligent2ndsubstationSmartMeterAMIElectricalVehiclesLegacyFieldSystemsPartnerDataExternalDataSourcesSocialMedia,etc.
ExternalCloud(PublicorPrivate)DataSubscriberDataSubscriberServicesDataasaserviceConsumerAnalyticsOperationalOpenDataDashboardforBIExternalAnalyticsRegulatoryComplianceDemandResponseProgramCapacityPlanning,LoadForecastLoadBalancerAPIExposureFigure4:Schemaofanenergydatasuperstore9withgreaterprecision,basedonanunderstandingofdemandandpricesorpredictpotentialoutagesandequipmentfailuresandtakeimmediatepreventativeaction.
Havingabetterinsightintohowmuchenergywillberequiredinaparticularlocationenablesutilitiestomoreeffectivelyplanforgenerating,buying,ordistributingelectricitytothatlocation.
Havinganunderstandingofenergyconsumptionandrenewableenergyinjectionatthesubstationleveloffersalevelofinsightsimilartothatfromsmartmetersinhundredsofhomes,butatlowercost.
ConclusionItisalmostimpossibletoexaggeratethetransformationaleffectofthesmartgridonenergygeneration.
Itreleasesvaluabledatafromeverypointofthephysicalinfrastructure,andprovidesthemechanismswherebythatdatacanbeconvertedintoextraordinaryinsightandunderstandingintoeveryaspectofthebusiness.
Utilitiesandsystemoperatorshavethepotentialtobecomedatapowerhouses:processinggigabytesofdataasgigawattsofpowertraversethenetwork.
Butifthispotentialistoberealized,buildingthenewdata-drivenoperationmuststartnow.
Asthispaperhasdemonstrated,thevolume,varietyandvelocityofdatainvolvedpresentssignificantchallenges:notjusttothosewhomustdesignthereferencearchitectureforhandlingandanalyzingdata,butthoseinchargeofprotectingandsecuringitinthefaceofincreasedthreatlevels.
Thedatasuperstorepresentedhererepresentsakeybuildingblockforthenewsmartgrid–andoneforwhichthetechnologyandcapabilitytobuildisalreadyavailable.
Enablingdatatobecapturedandanalyzed,queriedinrealtimeifnecessary,andcombinedflexiblytodeliveruniquenewinsights,itremovestheneedtodevelopdifferentarchitecturesforthevariousdifferentdatatypes.
Basedonopensourcecomponentswhereavailableitbuildsinsecurityandinteroperabilityateverylayer.
Withthedatasuperstoreinplace,utilitieswillbeabletodevelopnewinformation-driven,value-addedbusinessservices,aswellasdeployingthepredictive,proactiveandpreventiveanalyticsthatwilldrivetechnical,operationalandenergyefficienciesthroughoutthegrid.
www.
marketsandmarkets.
com/Market-Reports/smart-grid-technology-application-market-453.
htmlgclid=CP660aHV7L4CFaXHtAod-1MA4Awww.
greentechmedia.
com/articles/read/smart-grid-market-to-surpass-400-billion-worldwide-by-2020www.
mcafee.
com/ca/about/news/2015/q1/20150304-01.
aspxwww.
intel.
com/content/www/us/en/it-management/intel-it-best-practices/security-business-intelligence-siem-video.
htmlThesesecurityfeaturesarebasedontheopensourceprojectRhino,whichisavailablefrommanagedopensourceHadoopvendorssuchasClouderaInteltechnologies'featuresandbenefitsdependonsystemconfigurationandmayrequireenabledhardware,softwareorserviceactivation.
Performancevariesdependingonsystemconfiguration.
Nocomputersystemcanbeabsolutelysecure.
Checkwithyoursystemmanufacturerorretailerorlearnmoreatwww.
intel.
comIntel,theIntellogo,IntelXeon,IntelIoTGateway,IntelMashery,andIntelAES-NIaretrademarksofIntelCorporationintheU.
S.
andothercountries.
*Othernamesandbrandsmaybeclaimedasthepropertyofothers.
2015,IntelCorporationPleaseRecycle
易探云怎么样?易探云是国内一家云计算服务商家,致力香港服务器、国内外服务器租用及托管等互联网业务,目前主要地区为运作香港BGP、香港CN2、广东、北京、深圳等地区。目前,易探云推出深圳或北京地区的适合挂机和建站的云服务器,国内挂机宝云服务器(可选深圳或北京地区),独立ip;2核2G5M挂机云服务器仅330元/年起!点击进入:易探云官方网站地址易探云国内挂机宝云服务器推荐:1、国内入门型挂机云服务器...
快快CDN主营业务为海外服务器无须备案,高防CDN,防劫持CDN,香港服务器,美国服务器,加速CDN,是一家综合性的主机服务商。美国高防服务器,1800DDOS防御,单机1800G DDOS防御,大陆直链 cn2线路,线路友好。快快CDN全球安全防护平台是一款集 DDOS 清洗、CC 指纹识别、WAF 防护为一体的外加全球加速的超强安全加速网络,为您的各类型业务保驾护航加速前进!价格都非常给力,需...
介绍:御速云成立于2021年的国人商家,深圳市御速信息技术有限公司旗下品牌,为您提供安全可靠的弹性计算服务,随着业务需求的变化,您可以实时扩展或缩减计算资源,使用弹性云计算可以极大降低您的软硬件采购成本,简化IT运维工作。主要从事VPS、虚拟主机、CDN等云计算产品业务,适合建站、新手上车的值得选择,拥有华东江苏、华东山东等国内优质云产品;香港三网直连(电信CN2GIA联通移动CN2直连);美国高...
javlibrary.com为你推荐
杨紫别祝我生日快乐关于“致自己生日”的唯美句子有哪些?甲骨文不满赔偿公司倒闭员工不满一年怎么赔偿月神谭有没有什么好看的小说?拒绝言情小说!丑福晋大福晋比正福晋大么seo优化工具seo优化软件有哪些?抓站工具一起来捉妖神行抓妖辅助工具都有哪些?ip查询器怎么样查看自己电脑上的IP地址bbs2.99nets.com天堂1单机版到底怎么做partnersonline国内有哪些知名的ACCA培训机构www.ijinshan.com金山毒霸的网站是多少
北京虚拟主机租用 主机屋 a2hosting 台湾服务器 腾讯云数据库 免费网站监控 ixwebhosting php免费空间 panel1 国外在线代理 福建天翼加速 国外免费全能空间 河南m值兑换 网站卫士 vip购优惠 如何注册阿里云邮箱 国外网页代理 广东服务器托管 服务器托管价格 镇江高防服务器 更多