treatmentsjavlibrary.com

javlibrary.com  时间:2021-03-19  阅读:()
ThechallengesforfutureenergysystemsDigitizationintheenergysectorcontinuesapace.
By2016,theglobalmarketforsmartgridtechnologies,whichincludessensors,managementandcontroltechnologies,communicationnetworks,andsoftware,willbeworth$80.
6billion:agrowthof28.
7%from2011.
By2020,theglobalsmartgridmarketisforecasttoexceed$400billion.
IntheEU,policiesareencouragingthedevelopmentofdecentralizedelectricitygenerationinwhichelectricExecutiveSummaryTheenergyindustryisanincreasinglydigitalindustry.
Boththeexternalmarketandinternalinfrastructurearebeingtransformedbytheemergenceofthesmartgrid.
Inthefuture,thegridwillsimplybeoneautonomousenergysystemsteeredbyanalytics:anexampleoftheInternetofThings(IoT)inaction.
Withend-to-endtransparencyofdistributionandtransmission,utilitiesandoperatorswillbebetterabletounderstandbothgridperformanceandcustomerbehavior.
ThatinsightcanbeusedtooptimizeOpExandCapExandcreatenewbusinessservices.
Thechallengewillbenotjusttogatherandsecuredatafromahugelydiverserangeofsources,butalsotomakesenseofawidevarietyofstructuredandunstructuredformats.
ThispaperconsidershowIoTtechniquesapplytoasmartgridenvironment,examinesthedatamanagement,analysis,andsecurityrequirementsandintroducestheconceptofa'datasuperstore'asthefoundationforsuccessfulgridinfrastructuresofthefuture.
vehicles,energystorageandflexibledemandareallexpectedtoplayasignificantrole.
Thisdecentralizedvision,whichenablesbi-directionalflowofelectricity,isdependentonintelligentsystemsthatdeliverbi-directionalflowofinformationtosupportpredictablefunctionsandmonitoringcapabilities.
Inaddition,newvariableslikeunexpectedandmoreextremeweatherconditions,cyber-attacksDigitizingpowerutilitiesBusinesstransformationdrivenbyadvancedanalyticsChristianDonitzkyEnergyIndustrialSolutionArchitect,IntelEMEAOliverRoosEnergyBusinessDevelopmentManager,IntelEMEAParvizPeiraviPrincipalArchitect,IntelEnterpriseSolutionSalesSylvainSautySmartGridArchitect,IntelEMEAWhitePaperGridandhighlevelsofintermittentfeed-infromrenewablespresentachallengetosystemresilience.
Theriseofprosumers,inwhichordinaryenergyconsumersalsoproduceenergyfromsmalltomid-scaleinstallations,onlyaddstothechallenge.
Makinguseofthepotentialflexibilityofboththegridanditscustomerstoovercomeconstraintsandtooptimizeperformanceof,andinvestmentin,newandexistingnetworkassetsisincreasinglyimportant.
ThesmartsecondarysubstationThedisruptioncausedbymultipleandunpredictablesourcesofrenewableenergygenerationandthedecentralizationoftheenergyinfrastructurepresentsbothchallengesandopportunitiestoutilitiesandsystemoperators.
Distributionserviceoperators(DSOs)candevelopnewbusinessmodelsandservices,butmustreorganizetheiroperationsinordertodoso.
Somehavealreadystartedonthisprocessandareexamininghowbesttodevelopandoperatetheirnetworksinthelightofthesechanges.
Thekeypointsofconsiderationforthisreorganizationare:TheneedforclosercooperationwithTransmissionServiceOperators(TSOs)toestablishgridcodesandactivelymanageandoperateasmarternetworkTheneedtobalancegenerationandconsumptionatalocallevel,whilestillplanningoperationsinconjunctionwiththoseofTSOsEnsuringinfrastructurecanbeintegratedintoEuropeanplansfortrans-nationalinterconnectionandfuturenetworkoperationTheendgoalisa'transactionalenergysystem'inwhichdecision-makingprocessestakeplaceinrealtimethankstohigh-performancedataaggregationandprocessing.
Suchatransactionalsystemrequireseffectiveworkflowmanagementandprocessesforconfiguring,switchinganddispatching,aswellasanefficientcommandandcontrolresponsesystem.
Underpinningallthisareappropriatelevelsofcyber-securityneededtoprotectcriticalinfrastructure.
Inotherwords,itneedstobe'smart'.
Thesecondarysubstationisagoodillustrationofthissmartsysteminaction.
Thetraditionalenergygridisbasedonthepremisethatpowerisgeneratedataremotepowerplantandtransmittedtowardsdomestic,commercialandindustrialconsumers.
Inthismodel,thesubstationmerelyconvertsmediumvoltagetolowvoltageanddistributesittoalimitednumberoflocalusers.
However,thearrivalofprosumersandtheirvarious,unpredictablerenewablegenerationsources,invertsthatmodelasenergyisfedbackintothegridatvariouspointsacrossit.
Inthismodel,thesecondarysubstationisamuchmorecomplexinterfacebetweentheDSO,itsconsumers,anditsprosumers.
Toperformthisnewrole,thesecondarysubstationneedstobeequippedwithsensing,communicationandcomputepoweruptoandincludingedgeanalyticsfunctions.
Thesmartgrid,dataandtheIoTThepropertiesofthesmartgridaretypicalofanInternetofThings(IoT)deployment.
AnIoTimplementationconsistsofconnecteddevices,asensornetwork,agatewayforaggregatingandTableofContentsExecutiveSummary1Thechallengesforfutureenergysystems1Thesmartsecondarysubstation.
2Thesmartgrid,dataandtheIoT.
.
.
4Fromreactivetoproactiveanalytics.
4Topologiesfordataflow.
5Securityinthecriticalinfrastructure.
5Deployingadataarchitectureforthesmartgrid5Datacollectionandmessagetransfer6Datastorage:theadventofthedatasuperstore6Eventstreamprocessing(ESP)6DistributingdatausingAPIs7Fromrawnumberstobusinessinsight.
7Conclusion.
72AnalyticsServicesConsumerAnalyticsEventAnalyticsOperationalAnalyticsFinancialandBusinessAnalyticsReportingDataAnalysisStatisticalAnalysisAppliedMachineLearningDataMiningTimeSeriesAnalysisDataVisualizationGraphAnalyticsDataArchitectureDataStagingDataDiscoveryDataModelingModelValidationDataCurationDataEngineeringDataCollectionandIntegrationDataStorageDataCleansingDataQualityDataIntegrityDataClassificationCallDataRecord(CDR)EventDataTimeSeriesDataOperationalDataMetaDataDataSourcesTransmissionLineSubstationAdvancedMetering(AMI)EngineeringThirdPartyWeather,Twittertransmittingdata,andaprivateorpubliccloud–allconnectedthroughawiredorwirelessnetwork.
Wherenewdevicesareconnected,gatewayfunctionalitycanbebuiltinsothatdataflowsremainthesame.
LikeotherIoTimplementations,thevalueofthesmartgridlieslargelyindataitproducesandtheanalysisthatitenables.
Intheexampleofthesecondarysubstation,onesubstationproducesarelativelysmalldataset:thecurrentontheprimaryandsecondaryfeeders;voltageandcurrentontheprimaryandsecondarysideofthetransformer;thetransformer'sinternaltemperature;andrealandreactivepowerindicators–whichcanhelptotracetherenewablesinjectionandmaintainrightvoltagelevelalongtheline.
However,whenthatismultipliedoverseveralhundredsubstationsitbecomesaverysubstantialdataset.
Onpaper,thearchitectureneededtoreleaseandusethisdatafromacrosstheinfrastructurelooksrelativelystraightforward.
Butoncewelookbeyondthesubstationtothewiderinfrastructure,thenumberandvarietyofdevices,frompowerplanttotransformers,transmissionanddistributionsystems,andsmartmetersatusers'premises,createanumberofspecificchallenges,namely:Designinganeffectivedatanetworkformultipledatatypes,sourcesandtreatmentsEnablingadvancedanalyticsonawidevarietyofdatasetsandsubsetsandwithindifferingtimeframesSecuringdataandcommunicationsinfrastructureinthefaceofincreasedthreatlevelsThedesignofthearchitecturealsoneedstotakeintoaccountavarietyofpotentialusecases.
Again,thesubstationisastartingpointandas'smart'capabilitiesscaletomoreFigure1:Analyticscapabilityframeworkdevicesanddifferenttypesofdevices,moredatawillbeproduced.
Figure1givessomeexamplesofwhatcanbeachievedwiththenewanalyticscapability.
Thebottomrowshowsjustsomeofthepotentialdatasourcesinthissmartenvironment:fromtransmissionlinestoexternalsourceslikeweatherreportsandevensocialmedia.
Throughtheapplicationofvariousprocesses,anumberofvalue-addedbusinessservicesaremadepossible.
Takingadvantageoftheincreasedinsightthisdataproduces,thesenewbusinessservicesandfunctionscanbebasedonconsumerbehavior,forexample,orinsightintooperationalperformance.
Operatorsandutilitieslookingtoaddsmartcapabilitiestotheirinfrastructurecanstartwiththeirchosenusecaseandthenestablishthenecessarydatasourcesanddataprocessingfunctionstodeliverit.
3Figure2showsagenericframeworkforasmarterenergysolutionandtheframeworkanalyticsthatareneededtosupportcurrentandfuturebusinesscases.
ItillustrateswhereinformationflowsfromthesubstationtotheDSOandontotheTSO,aswellastheflowbetweentheseentitiesandrenewableenergysources(RES).
Toensurethesuccessfuloperationofthissmartframework,transparencyacrosstheoperationallevelsanduptotheTSOisessential.
Inaddition,communicationandGridCodestandardsareneededtoenableseamlessdatatransmissionfromthesensortothedatamanagementsystem.
FromreactivetoproactiveanalyticsTheusecasesenabledbythesmartgriddependonricherdatasets,greateranalyticscapabilities,andnewformsofanalysis.
Whereastoday'sdatamanagementandcontrolsystemsareretrospectiveandlookatwhathashappenedandwhy,futuresystemswillallowutilitiestopredictproblemsandsotakepre-emptiveactiontoavoidthem.
Ifwegobacktotheexampleofthesecondarysubstation,themonitoringsystemscurrentlyinplacemightobserveafailureanddiagnosethataswitcherisblocked.
Inthefuture,amoresophisticatedanalyticscapabilitywouldallowtheoperatortogobeyondthisdescriptiveresponseandenableamoreproactiveandpredictivecapability.
Assystemsbecomemoreadvanced,wecouldthereforesee:PredictiveanalyticsmodelingfutureloadsothatcriticalpatternscanbeanticipatedbeforetheyhappenPrescriptiveanalyticstriggeringamaintenanceteamtoservicebeforeaminorproblembecomesacriticalsituationProactiveanalyticsenablingDSOstoenhancetheirservicetoTSOsbyprovidinginsightintoconsumerbehavior,onwhichmoreappropriatecontractsandservicescanbebasedThistransformedanalyticscapabilitywillenableoperatorstorespondtoproblemsimmediately,toplanenergydistributioninnear-realtime,andtomanagethegrid'shealthandenergygenerationinthelongerterm.
TopologiesfordataflowToensurethesepotentialbenefitsarerealized,utilitiesandsystemoperatorsneedtodesigndataflowandanalyticsappropriately.
InaccordancewithIoTdesignprinciples,therearethreemaintopologiesfordataflowandanalyticsprocessing:Cloudanalytics:inusecaseswherelatencyandresponsetimearenotcriticalfactors,adirectconnectionfromthedevicetothecloudenablesanalyticstobeperformedinthecloud.
Thisismostsuccessfulwhenlowvolumesofdataareinvolvedandthecommunicationsnetworkdoesnotbecomeoverloaded.
Someactivities,modBusmodBus/TCPIEC60870OPC-UA/FTPIEC61850PredictiveAnalytics(IntelServer,CentralAnalytics)APISCADA-SystemGridoperator-Opscenteri2SubStIntelligentSecondarySubstation(IntelHardware,IntelCorei5,EdgeAnalytics)PredictiveAnalyticsinput/outputdataPhysicalMeasurementFeeder+TransformerWeatherDataProsumerIndependentSolarGenerationI/OAggregator(IntelQuark)modBus/TCPI/OI/OI/Oi2SubStOperationalCenterofDSOSecondarysubstationOperationalDataofTSOFigure2:Thegenericframeworkforasmarterenergysolutionacrossfunctionalvoltagelevels4suchasbillingandcustomermanagement,arelikelytoremaincentrallymanagedandsoapplicationsthatbuildonthemwillalsomostlikelyneedtoberuninthecloud.
Balancedarchitecture:whenareal-timeresponsetoasimpleeventisneeded,suchasremoteactivationorshut-down,theinfrastructuredesigncallsforahybridtopologywherethesensor,actuatororgatewaycanprovidesimpleanalyticssuchasfilteringdataordetectinganomaliesinrealtime.
Thisarchitectureenablestheoperatortotakeimmediateactionatthedevicelevelandreducesdependenceontheresponsivenessandavailabilityoftheconnectiontothecloud.
Thegatewaycanalsosendbatchdatatoanintermediate'sensorcloud'forfurtheranalysis.
Severalsecondarysubstationsinthesamegeographicareacouldconnecttothesamesensorcloudtocommunicatewitheachother,forexample,withthesensorcloudsendingaggregatedatabacktothepublic/privatecloudforfurtherlonger-termanalysis.
Edgeanalytics:whenthereneedstobeareal-timeresponsetoacomplexevent,orthereisextremelylimitedbandwidthtotransferdatainrealtime,asystemthatcanperformcomplexanalyticsattheedgeispreferable.
Inthiscase,thesensor,actuator,device,orgatewayanalyzesdataautonomouslyandconnectstotheback-endcloudwheneverthetransferofbatchdataispossible.
Trendanalysisonlargeraccumulateddatasetsisperformedbytheback-endcloudandtheresultscandirectlyorindirectlychangethewaythedevicesoperate.
Thismodelisalsoappropriateforstreamliningnon-criticaldatabeforeuploadingittothecloud.
Forexample,anaggregatorofsmartmeterscouldprocessdataandsummariesofneighborhoodenergyuseina15-minuteperiod,andsendthesummarytothecloud.
Itisimportanttonotethatthereisastrongcaseforaddingmoreintelligencetosensors,actuators,anddevicestosimplifytheend-to-endinfrastructureandreducetheneedforthecostlytransferofhugeamountsofdatatothecloud.
SecurityinthecriticalinfrastructureTheneedtosecuretheinformationnetworkalongsidetheenergytransmissionnetworkisakeychallengewhenbuildingthesmartgrid,andisadeterminingfactorindatatopologydesign.
Withmachinelearningandpredictiveanalyticscomingtothefore,bothdevicesanddatarequireprotection.
So,likeanyotherexampleoftheIoT,systemdesignersneedtoensurethatallcommunicationsbetweendevicesandbetweenthegridandthecloudaresecureandcomplywithregulatoryrequirements–withoutimpedingdataflow.
Sinceexistinginfrastructurewillneedtobeprotectedalongsidethenew,includingthatwhichisnormallynot'touched'bysystemoperators,theutility'sworkflowandprocessesfortheemergingsmartgridwillneedtobedesignedtoensuretheappropriatelevelsofsecuritycanbeensured.
Thiscanbeachievedinpartthroughthedistributionofintelligenceinthesmartgrid.
Thedecentralized,bi-directionalnatureofthesmartgridmeansthatasecuritygatewaycanbeinstalledateachdataentrypoint.
Thiscanactasafirewallwhileanonymizingandencryptingsensitivedataatrestandinmotion.
However,developingatrulyend-to-endsecuritysolutionrequiresthecontributionofhardwaremanufacturersandsoftwaredevelopersandtheirabilitytocreatesecuresolutionsthatenableandprotectdataflowsandsystemintegrity.
Interoperabilitywillbeessential.
Solutionsareavailablethatenableoperatorstoimplementextendableandadaptablesecuritymeasurestoaccommodaterapidlygrowingdatavolumesandtheexpandinganalyticsenvironment.
Achievingfullsituationalawarenessacrossalldomainsofthesmartgridtodeterminewhetheranattackisinprogressisakeypriority.
TheIntelSecurityCriticalInfrastructureProtection(CIP)technologyplatformsecureslegacysystemswithinthegridaswellasnewcapabilitiesastheyareadded.
Asecuremanagedplatform,itincludesfundamentalbuildingblocksforprotectinggridinfrastructuretailoredtomachine-to-machineenvironments.
Theseincludedeviceidentity,malwareprotection,dataprotection,andresilience.
Asecurityinformationandeventmanager(SIEM)liketheIntelITSecurityBusinessIntelligenceArchitecturecanalsobeintegratedtothedatastoretobringthefullreal-timevisionandsituationalawarenessthatisrequiredtooperateasecuresmartgrid.
Itisequallyimportanttopayattentiontosecurityofthedataplatforminfrastructureandallitscomponentparts.
Forexample,whereaHadoop*5clusterisusedforstoringandprocessingdata(seepage8),componentssuchasHive*,HBase*,ClusterManagement,theHadoopfilesystem(HDFS),andfilesmustbesecured.
HereIntelAES-NIsecurityaccelerationallowsfilestobeencryptedintheHDFS(whileatrest)andsecurescommunicationsbetweennodeswithineachHadoopcluster(wheninflight).
DeployingadataarchitectureforthesmartgridHavingestablishedpotentialusecases,theanalyticsrequirements,andsecuritydemandsoftheirsmartgrid,systemoperatorsneedtodevelopanappropriatearchitecture.
Inthissection,welookatwhatsuchaninfrastructuremightlooklike.
Atadetailedfunctionallevel,thereareanumberofessentialrequirementsfortheinfrastructure,including:Theabilitytocommunicatewithavarietyofdiversedevices,plussupportformultiplecommunicationsprotocolsSupportformultipledatamodels,includingIEC61850forexchanginginformationaboutmediumandlowvoltageelectricitydistributionandtheCommonInformationModel(CIM)forexchanginginformationaboutassetsbetweenapplicationsSupportformulti-applicationandmulti-tenantenvironmentssothatdatacanbeusedfordiversebusinesspurposesCloud-baseddeliverytoensurethatsystemscanscaleondemandandwithstandfailureSupportformodularandopen-architecturephilosophy,includingtheuseofopen-sourcesolutionswhereappropriateTheabilitytocaterforgatheringandstoringdataforanalysis,aswellasexposingdatatootherapplicationsIntegrationwithexistinginfrastructureandapplicationsImportantly,datastorageandanalyticalcapabilitiesmustbeabletohandlestructured,semi-structuredandunstructureddataandcombineitwhereappropriate.
Incontrasttostructureddata,whichistypicallyVisualAnalyticApplicationODBC,JDBCTCP/IPODBCHadoopClusterSparkSparkHBaseHBaseHDFSHDFSHDFSHDFSMapreduceExternalDataSourcesWeather,SocialMedia,etc.
UtilityApplicationsGIS,CIS,MDM,DREnterpriseApplicationsSCM,CRM,ERP,BPM,etc.
InMemoryDB/AnalyticEngineODBC,JDBC,JSONEventMessagingInfrastructureEnterpriseServiceBusWiredandWirelessNetworkSmartGridInfrastructureDeviceintelligentsubstationSensorHubSensorHubMessageBrokerLoadBalancerStreamEventProcessing(SEP)NotificationCommandandControlBusinessProcessManagementBusinessInteligentAdvancedAnalyticsDatawarehouseFigure3:Proposedhybridarchitectureforanenergydatasuperstore6sourcedfromenergymanagement(EMS),distributionmanagement(DMS),ormeterdatamanagement(MDM)systems,unstructureddataincludeslessformalsourcessuchasvideoandaudiosystemsusedtoremotelymonitorthehealthandsecurityofgridassets.
Semi-structureddatafallsbetweenthetwoandcanincludedeviceconfigurationfilesinXML,amongothers.
Asthepatchworkofdatagatheredacrossthegridbecomesmorecomplex,thiswillbeadefiningfeatureofasuccessfularchitecture.
Theotherdefiningfeatureofthesmartgridisthatdataanalysisneedstobeperformedinrealtime,near-realtime,asabatchprocess,andduringstreaming.
Forexample,inaSCADAsystem,real-timedatacouldbeprocessedwithlessthanfoursecondsoflatency.
Batchprocessingcouldbeappliedtosmartmeterdatausedinbilling,whilestreaminganalysiscouldbeusedtocontinuouslymonitorthehealthandsecuritystatusofthegridinfrastructure.
Withtheseneedsinmind,thearchitectureshouldconsistofdatacollectors,aneventmessaginginfrastructure,persistentstorage,dataprocessing,appliedmachinelearninganddatamining.
ThisisshowninFigure3.
Thisinfrastructuremayalsoincludeeventstreamprocessing(ESP),advancedanalyticsusingin-memoryappliances,andanenterpriseservicebus(ESB)toenableapplicationstoexchangedatawitheachother.
Althoughthesesolutionscanenableutilitiestobuildplatformsmorequickly,integrationofdifferentcomponentsmayprovechallenging.
Apackagedsolution,basedonproprietaryoropen-sourcetechnologiesfromdifferentvendorssuchasMicrosoftAzure*andCloudera*EnterpriseDataHub,mayreducethiscomplexity.
DatacollectionandmessagetransferAswithanydistributedIoTenvironment,communicationonthesmartgridinvolvesmessagesbeingpassedbetweenvariousdevicesandnetworknodes.
Thismessage-centricapproachcantakemanyforms,fromsimpledirecttransmissionstomorecomplexmessagequeuesandtransactionalsystems.
Inallofthem,theunitofinformationexchangeisthemessageitself:theinfrastructure'sroleistoensurethatmessagesgettotheirintendedrecipients.
Amessageprocessinginfrastructureforthesmartgridshouldofferthefollowing:Cross-platforminteroperabilityDistributed,looselycoupledarchitecturethatiseasytoscaleandmanageLowlatencyandhighthroughputforpublishingandsubscribingtomessagesGuaranteedmessagedeliveryAdvancedfilteringandqueryingformessagesSupportformultiplesubscribersAutomaticloadbalancingtopreventcriticalgridconstellationsSupportforbothbatchandreal-timestreamingapplicationsMaturityandproductionreadinesswithsupport,maintenance,andcomprehensivedocumentationSupportforcommonapplicationdevelopmentenvironments(suchasScala*,Java*,andPython*)ReducednumberofserversinthedatacenterOpen-sourcebasedoptionsforanevent-messaginginfrastructure(EMI)includeKafka*,RabbitMQ*,ActiveMQ*,ZeroMQ*,JoramMQ*,HornetQ*,andDIPQ*.
Selectionagaindependsonthebusinessusecaseaswellastechnicalrequirements,forexample,theneedforsub-secondresponsetimes.
Wherethereisaneedforhigh-throughput,low-latencyconnectivitythroughwhichhundredsofmillionsofeventsaretransmittedpersecond,Kafkaisregardedastheplatformofchoice.
Itssupportforbatchandstreamingservices,andabilitytoholdanddistributelargevolumesofmessagesareimportantfeatures.
TheIntelIoTGatewayintegratestechnologiesandprotocolsfornetworking,embeddedcontrol,enterprise-gradesecurity,andeasymanageability,onwhichapplication-specificsoftwarecanrun.
Italsoenablesseamlessandsecuredataflowbetweendevicesandthecloud.
ByusingtheIntelIoTGatewaytogatherdata,operatorscantakeadvantageofpre-integrated,pre-validatedhardwareandsoftwarebuildingblockstoconnectlegacyandnewsystems.
Datastorage:theadventofthedatasuperstoreAmodernplatformabletoperformbig-dataanalyticsisanessentialcomponentofthesmartgrid.
Thedatasuperstorearchitectureprovidesaplatformforanalyticsthatenables7utilitycompaniestocollectdisparatedatasourcesandeffectivelyturnthemintobusinessinsight.
Suchaplatformcanbebuiltusingthreekeyelements:Anenterprisedatawarehouse(EDW)forinteractivequeryingofstructureddataAnApacheHadoopclusterforstoring,processingandanalyzingpoly-structureddataincludingbatch,near-realtimeandstreaminganalyticsAnin-memoryanalyticssolutiontoprovidereal-timeanalysisofdatasets,particularlythemostvaluableandsensitivesubsetsofdatastoredintheEDW.
SystemssuchasOracleExalytics*,SAPHANA*andIBMNetezza*,whichcanbebasedontheIntelXeonprocessorE7productfamily,allperformthistaskLinkingtheEDWandHadoopclustermakesitispossibletoaddressdiverseusecaserequirements.
Hadoopexcelsasahigh-speed,massivelyscalableextract,transform,andload(ETL)solution,thatcanprocesspoly-structureddata.
Theprocesseddatacanbefurtheranalyzedbynewandexistingapplications,suchasbusinessintelligence,deeplearningandmachinelearning,tosupportinteractivequeriesandotheradvancedneeds.
Withitsdistributed,parallel-processingcapabilities,theHadoopclustercanrapidlygather,storeandprocesspetabytesofpoly-structureddatabycoordinatinglocalstorageandcomputationacrosstens,hundreds,oreventhousandsofservers.
Eachserverstoresandprocessesasubsetofthedataand,becausetheapplicationsexecuteinparallel,performanceandcapacitycanscalewitheachserverthatisaddedtothecluster.
TheHadoopframeworkincludesavarietyofcomponentsformanagingdataandapplications,including:HadoopDistributedFileSystem(HDFS):afaulttolerantandself-healingdistributedfilesystemdesignedspecificallyforlarge-scaledataprocessingworkloadswherescalability,flexibility,andthroughputareessentialrequirementsMapReduce*(MR):amassivelyscalable,paralleldata-processingsoftwareframeworkthatworksintandemwithHDFSforcondensinglargevolumesofdataintousefulaggregatedresultsHBase,Cassandra*andotherNoSQLdatabases:runontopofaHadoopclusteroronaseparatecluster,thesecanextendthecapabilitiesofHadoopHive:adatawarehousesystemforHadoopthatfacilitatesdatasummarization,adhocqueries,andtheanalysisoflargedatasets,HiveprovidesamechanismforaccessingdatafromHDFSandforqueryingthedatausingaSQL-likelanguage(HiveQL)Mahout*:adata-mininglibrarythatprovidesalgorithmsforclustering,collaborativefiltering,regressiontesting,andstatisticalmodelingEventstreamprocessing(ESP)Streaminganalysisisappropriatewhenthereisacontinuousflowofdata,suchasinformationfromadvancedmeteringinfrastructure(AMI)ormeteorologicalandatmosphericreports,thatneedstobeanalyzedasitarrives.
Inadditiontocommerciallyavailablesoftware,open-sourceapplicationssuchasthein-memorySpark*andSparkStreaming*computingframeworksupporteventstreamprocessingandcanbeusedtoidentify,filter,andprocesstargetedinformation.
Theysharethesameprogramminglanguageandaframeworkthatsupports'exactlyonce'messagedeliverytoeliminatemessageloss.
SparkStreamingenablesdeveloperstowritestreamingapplicationsforthecontinuousprocessingofmicro-batchesinthesamewayaswritingbatchprocessingprogramsforSpark.
Thissimplifiesapplicationdevelopmentandgivesdatascientiststheframeworktoprovidecomprehensiveviewsbasedonreal-timeandhistoricaldata.
BothSparkandSparkStreamingleveragetheHadoopdistributedarchitectureandcanbesupportedasstandalonesolutionsorintegratedinaHadoopsolution.
DistributingdatausingAPIsProvidingaconsistentwaytoaccessandquerydataandthenexposeittoothertrustedapplicationswithoutpoint-to-pointintegration,APIsarepowerfulandflexibletoolsforintegratingandsharinginsightintobusinessprocesses.
Asaresult,theycanshortenthetimetomarketfornewsolutions,makingthemanimportantelementinthedevelopmentofdata-enabledusecasesandbusinessservices.
DemandresponseisoneexampleofhowAPIscandelivervaluethroughthesmartgrid.
Autilitycanorchestrateitsprocessing,networkandstorageresourcestoingestdifferentkindsofdata–forexamplefromsolar8photovoltaic(PV)systemsorSCADAcontrols–withdifferentlevelsoflatency.
AbrokerordispatcherwouldthentransferthedatatoanESPengineandHadoopclusterforreal-timeandbatch-orientedanalysisusingadvancedtechniques,suchasmachinelearning,forpatternandanomalydetection.
TheAPIlayerwouldthenexposetheprocesseddatatothenewgenerationofservices.
Alltheresourcesrequiredtoingest,process,analyze,anddelivertheresultingservicetobusinessuserscanbeorchestratedineitherapublicorprivatecloud.
Figure4providesahigh-levelviewoftherelevantarchitecture,fromdatasources(ontheleft)throughtobusinessservices(ontheright).
APIscanbeimplementedbyusingAPImanagementsolutionssuchasthosefromIntelMashery.
Aswiththedataitself,itisimportanttomanageandsecureAPIscentrallytoprovideflexiblebutcontrolledaccesstoinformationandresources.
OtherapplicationsthatcanbeexposedthroughanAPIinclude,butarenotlimitedto:regulatorycompliancereporting,businessintelligence,capacityplanning,consumeranalytics,andmash-upservices.
FromrawnumberstobusinessinsightAswehaveseen,thedevelopmentofthesmartgridisdrivenbyanumberofinternalandexternalfactors.
Asthispapersuggests,theadvantagesforutilitiesandoperatorsarethenewbusinessservicesthatareenabledbyinsightgainedfromadvancedanalytics.
Thedatasuperstorearchitectureprovidestheplatformforthislevelofanalyticsandenablesutilitycompaniestogatherdisparatedatasetsandturnthemintobusinessinsight.
Inanyanalyticsproject,theclaimisthat80percentofthetimeisspentondatapreparationandonly20percentisspentonmodeldevelopment,trainingandvalidation.
Thebigdatatechnologiesoutlinedherenotonlyprovidethescalabledatastorageandprocessingcapabilities,theygivedatascientistsdirectaccesstoentiredatasets–andsoacceleratedataanalysis.
Asaresult,datascientistscanrunconcurrentanalysisandsimulationswithamuchshortertimetocompletion.
Analyticsservicesthathavenotbeenviableuntilnow,suchasreal-timedetectionofanomaliesandcustomerbehavioralanalysis,arenowpossible.
Byincorporatingreadilyavailabledatafromexternalsources,utilitiesareabletoaddanotherlayerofinsightandpushfurtherintopredictiveandprescriptiveanalytics.
Forexample,theycanmanageenergyprocurementAnalyticCloudPlatformStreamEventProcessingCEPEngineEnterpriseDataHubHadoopClusterEnterpriseServiceBusDWODSEnterpriseApplicationsDataSourcesTSODSOintelligent2ndsubstationSmartMeterAMIElectricalVehiclesLegacyFieldSystemsPartnerDataExternalDataSourcesSocialMedia,etc.
ExternalCloud(PublicorPrivate)DataSubscriberDataSubscriberServicesDataasaserviceConsumerAnalyticsOperationalOpenDataDashboardforBIExternalAnalyticsRegulatoryComplianceDemandResponseProgramCapacityPlanning,LoadForecastLoadBalancerAPIExposureFigure4:Schemaofanenergydatasuperstore9withgreaterprecision,basedonanunderstandingofdemandandpricesorpredictpotentialoutagesandequipmentfailuresandtakeimmediatepreventativeaction.
Havingabetterinsightintohowmuchenergywillberequiredinaparticularlocationenablesutilitiestomoreeffectivelyplanforgenerating,buying,ordistributingelectricitytothatlocation.
Havinganunderstandingofenergyconsumptionandrenewableenergyinjectionatthesubstationleveloffersalevelofinsightsimilartothatfromsmartmetersinhundredsofhomes,butatlowercost.
ConclusionItisalmostimpossibletoexaggeratethetransformationaleffectofthesmartgridonenergygeneration.
Itreleasesvaluabledatafromeverypointofthephysicalinfrastructure,andprovidesthemechanismswherebythatdatacanbeconvertedintoextraordinaryinsightandunderstandingintoeveryaspectofthebusiness.
Utilitiesandsystemoperatorshavethepotentialtobecomedatapowerhouses:processinggigabytesofdataasgigawattsofpowertraversethenetwork.
Butifthispotentialistoberealized,buildingthenewdata-drivenoperationmuststartnow.
Asthispaperhasdemonstrated,thevolume,varietyandvelocityofdatainvolvedpresentssignificantchallenges:notjusttothosewhomustdesignthereferencearchitectureforhandlingandanalyzingdata,butthoseinchargeofprotectingandsecuringitinthefaceofincreasedthreatlevels.
Thedatasuperstorepresentedhererepresentsakeybuildingblockforthenewsmartgrid–andoneforwhichthetechnologyandcapabilitytobuildisalreadyavailable.
Enablingdatatobecapturedandanalyzed,queriedinrealtimeifnecessary,andcombinedflexiblytodeliveruniquenewinsights,itremovestheneedtodevelopdifferentarchitecturesforthevariousdifferentdatatypes.
Basedonopensourcecomponentswhereavailableitbuildsinsecurityandinteroperabilityateverylayer.
Withthedatasuperstoreinplace,utilitieswillbeabletodevelopnewinformation-driven,value-addedbusinessservices,aswellasdeployingthepredictive,proactiveandpreventiveanalyticsthatwilldrivetechnical,operationalandenergyefficienciesthroughoutthegrid.
www.
marketsandmarkets.
com/Market-Reports/smart-grid-technology-application-market-453.
htmlgclid=CP660aHV7L4CFaXHtAod-1MA4Awww.
greentechmedia.
com/articles/read/smart-grid-market-to-surpass-400-billion-worldwide-by-2020www.
mcafee.
com/ca/about/news/2015/q1/20150304-01.
aspxwww.
intel.
com/content/www/us/en/it-management/intel-it-best-practices/security-business-intelligence-siem-video.
htmlThesesecurityfeaturesarebasedontheopensourceprojectRhino,whichisavailablefrommanagedopensourceHadoopvendorssuchasClouderaInteltechnologies'featuresandbenefitsdependonsystemconfigurationandmayrequireenabledhardware,softwareorserviceactivation.
Performancevariesdependingonsystemconfiguration.
Nocomputersystemcanbeabsolutelysecure.
Checkwithyoursystemmanufacturerorretailerorlearnmoreatwww.
intel.
comIntel,theIntellogo,IntelXeon,IntelIoTGateway,IntelMashery,andIntelAES-NIaretrademarksofIntelCorporationintheU.
S.
andothercountries.
*Othernamesandbrandsmaybeclaimedasthepropertyofothers.
2015,IntelCorporationPleaseRecycle

DogYun香港BGP月付14.4元主机简单测试

前些天赵容分享过DogYun(狗云)香港BGP线路AMD 5950X经典低价云服务器的信息(点击查看),刚好账户还有点余额够开个最低配,所以手贱尝试下,这些贴上简单测试信息,方便大家参考。官方网站:www.dogyun.com主机配置我搞的是最低款优惠后14.4元/月的,配置单核,512MB内存,10GB硬盘,300GB/50Mbps月流量。基本信息DogYun的VPS主机管理集成在会员中心,包括...

Hostiger发布哥伦布日提供VPS主机首月七折优惠 月费2.79美元

Hostiger商家我们可能以前也是有见过的,以前他们的域名是Hostigger,后来进行微调后包装成现在的。而且推出Columbus Day哥伦布日优惠活动,提供全场的VPS主机首月7折月付2.79美元起的优惠。这里我们普及一下基础知识,Columbus Day ,即为每年10月12日,是一些美洲国家的节日,纪念克里斯托弗·哥伦布在北美登陆,为美国的联邦假日。Hostiger 商家是一个成立于2...

青云互联:美国洛杉矶CN2弹性云限时八折,15元/月起,可选Windows/可自定义配置

青云互联怎么样?青云互联是一家成立于2020年6月的主机服务商,致力于为用户提供高性价比稳定快速的主机托管服务,目前提供有美国免费主机、香港主机、香港服务器、美国云服务器,让您的网站高速、稳定运行。美国cn2弹性云主机限时8折起,可选1-20个IP,仅15元/月起,附8折优惠码使用!点击进入:青云互联官方网站地址青云互联优惠码:八折优惠码:ltY8sHMh (续费同价)青云互联活动方案:美国洛杉矶...

javlibrary.com为你推荐
地图应用看卫星地图哪个手机软件最好。access数据库access数据库的组成是什么bbs.99nets.com做一款即时通讯软件难吗 像hi qq这类的22zizi.com福利彩双色球22号开奖号同一ip网站同一个IP不同的30个网站,是不是在一个服务器上呢?www.e12.com.cn上海高中除了四大名校,接下来哪所高中最好?顺便讲下它的各方面情况m.2828dy.comwww.dy6868.com这个电影网怎么样?se9999se.comexol.smtown.comwww.kaspersky.com.cn卡巴斯基杀毒软件有免费的吗?稳定版的怎么找?yinrentangzimotang氨基酸洗发水的功效咋样?
免备案虚拟空间 域名服务dns的主要功能为 免费域名申请 中国万网域名 edgecast 网络星期一 新站长网 坐公交投2700元 稳定免费空间 电信主机 ca187 免费私人服务器 yundun 万网主机管理 免费外链相册 海外空间 ebay注册 东莞主机托管 工信部网站备案查询 免费稳定空间 更多