arXiv:adap-org/9806001v1
exhentai.org 时间:2021-04-06 阅读:()
3Jun1998NeuralNetworkDesignforJFunctionApproximationinDynamicProgrammingXiaozhongPangPaulJ.
WerbosAbstractThispaperwillshowthatanewneuralnetworkdesigncansolveanexampleofdicultfunctionapproximationproblemswhicharecrucialtotheeldofapproximatedynamicprogramming(ADP).
Althoughconven-tionalneuralnetworkshavebeenproventoapproximatesmoothfunctionsverywell,theuseofADPforproblemsofintelligentcontrolorplanningrequirestheapproximationoffunctionswhicharenotsosmooth.
Asanexample,thispaperstudiestheproblemofapproximatingtheJfunctionofdynamicprogrammingappliedtothetaskofnavigatingmazesingen-eralwithouttheneedtolearneachindividualmaze.
Conventionalneuralnetworks,likemulti-layerperceptrons(MLPs),cannotlearnthistask.
Butanewtypeofneuralnetworks,simultaneousrecurrentnetworks(SRNs),candosoasdemonstratedbysuccessfulinitialtests.
Thepaperalsoex-aminestheabilityofrecurrentneuralnetworkstoapproximateMLPsandviceversa.
Keywords:Simultaneousrecurrentnetworks(SRNs),multi-layerpercep-trons(MLPs),approximatedynamicprogramming,mazenavigation,neuralnet-works.
1Introduction1.
1PurposeThispaperhasthreegoals:First,todemonstratethevalueofanewclassofneuralnetworkwhichpro-videsacrucialcomponentneededforbrain-likeintelligentcontrolsystemsforthefuture.
Second,todemonstratethatthisnewkindofneuralnetworkprovidesbetterfunctionapproximateabilityforuseinmoreordinarykindsofneuralnetworkapplicationsforsupervisedlearning.
Third,todemonstratesomepracticalimplementationtechniquesnecessarytomakethiskindofnetworkactuallyworkinpractice.
X(t)Y(t)SupervisedLearningSystemActualY(t)^Figure1:Whatissupervisedlearning1.
2BackgroundAtpresent,intheneuralnetworkeldperhaps90%ofneuralnetworkapplica-tionsinvolvetheuseofneuralnetworksdesignedtoperformanceataskcalledsupervisedlearning(Figure1).
Supervisedlearningisthetaskoflearninganon-linearfunctionwhichmayhaveseveralinputsandseveraloutputsbasedonsomeexamplesofthefunction.
Forexample,incharacterrecognition,theinputsmaybeanarrayofpixelsseenfromacamera.
Thedesiredoutputsofthenetworkmaybeaclassicationofcharacterbeingseen.
Anotherexamplewouldbeforintelligentsensinginthechemicalindustrywheretheinputsmightbespectraldatafromobservingabatchofchemicals,andthedesiredoutputswouldbetheconcentrationsofthedierentchemicalsinthebatch.
Thepurposeofthisapplicationistopredictorestimatewhatisinthebatchwithouttheneedforexpensiveanalyticaltests.
Theworkinthispaperwillfocustotallyoncertaintasksinsupervisedlearning.
Eventhoughexistingneuralnetworkscanbeusedinsupervisedlearn-ing,therecanbeperformanceproblemsdependingonwhatkindoffunctionislearned.
Manypeoplehaveprovenmanytheoremstoshowthatneuralnetworks,fuzzylogic,Taylortheoriesandotherfunctionapproximationhaveauniversalabilitytoapproximatefunctionsontheconditionthatthefunctionshavecertainpropertiesandthatthereisnolimitonthecomplexityoftheapproximation.
Inpractice,manyapproximationschemesbecomeuselesswhentherearemanyinputvariablesbecausetherequiredcomplexitygrowsatanexponentialrate.
Forexample,onewaytoapproximateafunctionwouldbetoconstructatableofthevaluesofthefunctionatcertainpointsinthespaceofpossibleinputs.
Supposethereare30inputvariablesandweconsider10possiblevaluesofeachinput.
Inthatcase,thetablemusthave1030numbersinit.
Thisisnotusefulinpracticeformanyreasons.
Actually,however,manypopular2approximationmethodslikeradialbasisfunction(RBF)aresimilarinspirittoatableofvalues.
Intheeldofsupervisedlearning,AndrewBarron[30]hasprovedsomefunc-tionapproximationtheoremswhicharemuchmoreusefulinpractice.
Hehasproventhatthemostpopularformofneuralnetworks,themulti-layerpercep-tron(MLP),canapproximateanysmoothfunction.
Unlikethecasewiththelinearbasisfunctions(likeRBFandTaylorseries),thecomplexityofthenet-workdoesnotgrowrapidlyasthenumberofinputvariablesgrows.
Unfortunatelytherearemanypracticalapplicationswherethefunctionstobeapproximatedarenotsmooth.
Insomecases,itisgoodenoughjusttoaddextralayerstoanMLP[1]ortouseageneralizedMLP[2].
However,therearesomedicultproblemswhichariseineldslikeintelligentcontrolorimageprocessingorevenstochasticsearchwherefeed-forwardnetworksdonotappearpowerfulenough.
1.
3SummaryandOrganizationofThisPaperThemaingoalofthispaperistodemonstratethecapabilityofadierentkindofsupervisedlearningsystembasedonakindofrecurrentnetworkcalledsimultaneousrecurrentnetwork(SRN).
Inthenextchapterwewillexplainwhythiskindofimprovedsupervisedlearningsystemwillbeveryimportanttointelligentcontrolandtoapproximatedynamicprogramming.
Ineectthisworkonsupervisedlearningistherststepinamulti-stepeorttobuildmorebrain-likeintelligentsystems.
ThenextstepwouldbetoapplytheSRNtostaticoptimizationproblems,andthentointegratetheSRNsintolargesystemsforADP.
Eventhoughintelligentcontrolisthemainmotivationforthiswork,theworkmaybeusefulforotherareasaswell.
Forexample,inzipcoderecog-nition,AT&T[3]hasdemonstratedthatfeed-forwardnetworkscanachieveahighlevelofaccuracyinclassifyingindividualdigits.
However,AT&Tandtheothersstillhavedicultyinsegmentingthetotalzipcodesintoindividualdig-its.
ResearchonhumanvisionbyvonderMalsburg[4]andothershassuggestedthatsomekindsofrecurrencyinneuralnetworksarecrucialtotheirabilitiesinimagesegmentationandbinocularvision.
Furthermore,researchersinimageprocessinglikeLaveenKanalhaveshowedthatiterativerelaxationalgorithmsarenecessaryeventoachievemoderatesuccessinsuchimageprocessingtasks.
ConceptuallytheSRNcanlearnanoptimaliterativealgorithm,buttheMLPcannotrepresentanyiterativealgorithms.
Insummary,thoughwearemostinterestedinbrain-likeintelligentcontrol,thedevelopmentofSRNscouldleadtoveryimportantapplicationsinareassuchasimageprocessinginthefuture.
Thenetworkdescribedinthispaperisuniqueinseveralrespects.
However,itiscertainlynottherstserioususeofarecurrentneuralnetwork.
Chapter3ofthispaperwilldescribetheexistingliteratureonrecurrentnetworks.
Itwilldescribetherelationshipbetweenthisnewdesignandotherdesignsinthe3literature.
Roughlyspeaking,thevastbulkofresearchinrecurrentnetworkshasbeenacademicresearchusingdesignsbasedonordinarydierentialequa-tions(ODE)toperformsometasksverydierentfromsupervisedlearning—taskslikeclustering,associativememoryandfeatureextraction.
ThesimpleHebbianlearningmethods[13]usedforthosetasksdonotleadtothebestper-formanceinsupervisedlearning.
Manyengineershaveusedanothertypeofrecurrentnetwork,thetimelaggedrecurrentnetwork(TLRN),wheretherecur-rencyisusedtoprovidememoryofpasttimeperiodsforuseinforecastingthefuture.
However,thatkindofrecurrencycannotprovidetheiterativeanalysiscapabilitymentionedabove.
VeryfewresearchershavewrittenaboutSRNs,atypeofrecurrentnetworkdesignedtominimizeerrorandlearnanoptimaliterativeapproximationtoafunction.
ThisiscertainlytherstuseofSRNstolearnaJfunctionfromdynamicprogrammingwhichwillbeexplainedmoreinchapter2.
ThismayalsobetherstempiricaldemonstrationoftheneedforadvancedtrainingmethodstopermitSRNstolearndicultfunctions.
Chapter4willexplaininmoredetailthetwotestproblemswehaveusedfortheSRNandtheMLP,aswellasthedetailsofarchitectureandlearningprocedure.
ThersttestproblemwasusedmainlyasaninitialtestofasimpleformofSRNs.
Inthisproblem,wetriedtotestthehypothesisthatanSRNcanalwayslearntoapproximatearandomlychosenMLP,butnotviceversa.
Althoughourresultsareconsistentwiththathypothesis,thereisroomformoreextensiveworkinthefuture,suchasexperimentswithdierentsizesofneuralnetworksandmorecomplexstatisticalanalysis.
ThemaintestprobleminthisworkwastheproblemoflearningtheJfunctionofdynamicprogramming.
Foramazenavigationproblem,manyneuralnetworkresearchershavewrittenaboutneuralnetworkswhichlearnanoptimalpolicyofactionforoneparticularmaze[5].
ThispaperwilladdressthemoredicultproblemoftraininganeuralnetworktoinputapictureofamazeandoutputtheJfunctionforthismaze.
WhentheJfunctionisknown,itisatriviallocalcalculationtondthebestdirectionofmovement.
Thiskindofneuralnetworkshouldnotrequireretrainingwheneveranewmazeisencountered.
Insteaditshouldbeabletolookatthemazeandimmediately"see"theoptimalstrategy.
Trainingsuchanetworkisaverydicultproblemwhichhasneverbeensolvedinthepastwithanykindofneuralnetwork.
Alsoitistypicalofthechallengesoneencountersintrueintelligentcontrolandplanning.
Thispaperhasdemonstratedaworkingsolutiontothisproblemforthersttime.
Nowthatasystemisworkingonaverysimpleformforthisproblem,itwouldbepossibleinthefuturetoperformmanytestsoftheabilityofthissystemtogeneralizeitssuccesstomanymazes.
Inordertosolvethemazeproblem,itwasnotsucientonlytouseanSRN.
TherearemanychoicestomakewhenimplementingthegeneralideaofSRNsorMLPs.
Chapter5willdescribeindetailhowthesechoicesweremadeinthiswork.
Themostimportantchoiceswere:41.
BothfortheMLPandforthefeed-forwardcoreoftheSRNweusedthegeneralizedMLPdesign[2]whicheliminatestheneedtodecideonthenumberoflayers.
2.
Forthemazeproblem,weusedacellularorweight-sharingarchitecturewhichexploitsthespatialsymmetryoftheproblemandreducesdramaticallythenumberofweights.
Ineectwesolvedthemazeproblemusingonlyvedistinctneurons.
Thereareinterestingparallelsbetweenthisnetworkandthehippocampusofthehumanbrain.
3.
Forthemazeproblem,anadaptivelearningrate(ALR)procedurewasusedtopreventoscillationandensureconvergence.
4.
InitialvaluesfortheweightsandtheinitialinputvectorfortheSRNwerechosenessentiallyatrandom,byhand.
Inthefuture,moresystematicmethodsareavailable.
Butthiswassucientforsuccessinthiscase.
FinallyChapter6willdiscussthesimulationresultsinmoredetail,givetheconclusionsofthispaperandmentionsomepossibilitiesforfuturework.
2MotivationInthischapterwewillexplaintheimportanceofthiswork.
Asdiscussedabove,thepapershowshowtouseanewtypeofneuralnetworkinordertoachievebetterfunctionapproximationthanwhatisavailablefromthetypesofneu-ralnetworkswhicharepopulartoday.
Thischapterwilltrytoexplainwhybetterfunctionapproximationisimportanttoapproximatedynamicprogram-ming(ADP),intelligentcontrolandunderstandingthebrain.
ImageprocessingandotherapplicationshavealreadybeendiscussedintheIntroduction.
Thesethreetopics—ADP,intelligentcontrolandunderstandingthebrain—areallcloselyrelatedtoeachotherandprovidetheoriginalmotivationfortheworkofthispaper.
Thepurposeofthispaperistomakeacorecontributiontodevelopingthemostpowerfulpossiblesystemforintelligentcontrol.
Inordertobuildthebestintelligentcontrolsystems,weneedtocombinethemostsuitablemathematicstogetherwithsomeunderstandingofnaturalintelligenceinthebrain.
Thereisalotofinterestinintelligentcontrolintheworld.
Somecontrolsystemswhicharecalledintelligentareactuallyveryquickandeasythings.
Therearemanypeoplewhotrytomovestepbysteptoaddintelligenceintocontrol,butastep-by-stepapproachmaynotbeenoughbyitself.
Sometimestoachieveacomplexdicultgoal,itisnecessarytohaveaplan,thussomepartsoftheintelligentcontrolcommunityhavedevelopedamoresystematicvisionorplanforhowitcouldbepossibletoachieverealintelligentcontrol.
First,onemustthinkaboutthequestionofwhatisintelligentcontrol.
Then,insteadoftryingtoanswerthisquestioninonestep,wetrytodevelopaplantoreachthedesign.
Actuallytherearetwoquestions:51.
Howcouldwebuildanarticialsystemwhichreplicatesthemaincapa-bilitiesofbrain-likeintelligence,somehowuniedtogetherastheyareuniedtogetherinthebrain2.
Howcanweunderstandwhatarethecapabilitiesinthebrainandhowtheyareorganizedinafunctionalengineeringviewi.
e.
howarethosecircuitsinthehumanbrainarrangedtolearnhowtoperformdierenttasksItwouldbebesttounderstandhowthehumanbrainworksbeforebuildinganarticialsystem.
However,atthepresenttime,ourunderstandingofthebrainislimited.
Butatleastweknowthatlocalrecurrencyplayscriticalruleinthehigherpartofthehumanbrain[6][7][8][4].
AnotherreasontouseSRNsisthatSRNscanbeveryusefulinADPmath-ematically.
NowwewilldiscusswhatADPcandoforintelligentcontrolandunderstandingthebrain.
Theremainderofthischapterwilladdressthreequestionsinorder:1.
WhatisADP2.
WhatistheimportanceofADPtointelligentcontrolandunderstandingthebrain3.
WhatistheimportanceofSRNstoADP2.
1WhatisADPandJFunctionToexplainwhatisADP,letusconsidertheoriginalBellmanequation[9]:J(R(t))=maxu(t)(U(R(t),u(t))+)/(1+r)U0(1)whererandu0areconstantsthatareusedonlyininnite-time-horizonproblemsandthenonlysometimes,andwheretheanglebracketsrefertoexpectationvalue.
Inthispaperweactuallyuse:J(R(t))=maxu(t)(U(R(t),u(t))+)(2)sincethemazeproblemdonotinvolveaninnitetime-horizon.
InsteadofsolvingforthevalueofJineverypossiblestate,R(t),wecanuseafunctionapproximationmethodlikeneuralnetworkstoapproximatetheJfunction.
Thisiscalledapproximatedynamicprogramming(ADP).
ThispaperisnotdoingtrueADPbecauseintrueADPwedonotknowwhattheJfunctionisandmustthereforeuseindirectmethodstoapproximateit.
However,beforewetrytouseSRNsasacomponentofanADPsystem,itmakessensetorsttesttheabilityofanSRNtoapproximateaJfunction,inprinciple.
NowwewilltrytoexplainwhatistheintuitivemeaningoftheBellmanequation(Equation(1))andtheJfunctionaccordingtothetreatmenttakenfrom[2].
TounderstandADP,onemustrstreviewthebasicsofclassicaldynamicprogramming,especiallytheversionsdevelopedbyHoward[28]andBertsekas.
6Classicaldynamicprogrammingistheonlyexactandecientmethodtocom-putetheoptimalcontrolpolicyovertime,inageneralnonlinearstochasticenvi-ronment.
Theonlyreasontoapproximateitistoreducecomputationalcost,soastomakethemethodaordable(feasible)acrossawiderangeofapplications.
Indynamicprogramming,theusersuppliesautilityfunctionwhichmaytaketheformU(R(t),u(t))—wherethevectorRisaRepresentationorestimateofthestateoftheenvironment(i.
e.
thestatevector)—andastochasticmodeloftheplantorenvironment.
Then"dynamicprogramming"(i.
e.
solutionoftheBellmanequation)givesusbackasecondaryorstrategicutilityfunctionJ(R).
ThebasictheoremisthatmaximizingU(R(t),u(t))+J(R(t+1))yieldstheoptimalstrategy,thepolicywhichwillmaximizetheexpectedvalueofUaddedupoverallfuturetime.
Thusdynamicprogrammingconvertsadicultprob-leminoptimizingovermanytimeintervalsintoastraightforwardprobleminshort-termmaximization.
Inclassicaldynamicprogramming,wendtheexactfunctionJwhichexactlysolvestheBellmanequation.
InADP,welearnakindof"model"ofthefunctionJ;this"model"iscalleda"Critic.
"(Alternatively,somemethodslearnamodelofthederivativesofJwithrespecttothevariablesRi;thesecorrespondtoLagrangemultipliers,λi,andtothe"pricevariables"ofmicroeconomictheory.
SomemethodslearnafunctionrelatedtoJ,asintheAction-DependentAdaptiveCritic(ADAC)[29].
2.
2IntelligentControlandRobustControlTounderstandthehumanbrainscientically,wemusthavesomesuitablemath-ematicalconcepts.
Sincethehumanbrainmakesdecisionslikeacontrolsystem,itisanexampleofanintelligentcontrolsystem.
Neuroscientistsdonotyetun-derstandthegeneralabilityofthehumanbraintolearntoperformnewtasksandsolvenewproblemseventhoughtheyhavestudiedthebrainfordecades.
Somepeoplecomparethepastresearchinthiseldtowhatwouldhappenifwespentyearstostudyradioswithoutknowingthemathematicsofsignalprocessing.
Werstneedsomemathematicalideasofhowitispossibleforacomputingsystemtohavethiskindofcapabilitybasedondistributedparallelcomputation.
Thenwemustaskwhatarethemostimportantabilitiesofthehumanbrainwhichunifyallofitsmorespecicabilitiesinspecictasks.
Itwouldbeseenthatthemostimportantabilityofbrainistheabilitytolearnovertimehowtomakebetterdecisionsinordertobettermaximizethegoalsoftheorganism.
Thenaturalwaytoimitatethiscapabilityinengineeringsystemsistobuildsystemswhichlearnovertimehowtomakedecisionswhichmaximizesomemeasureofsuccessorutilityoverfuturetime.
Inthiscontext,dynamicprogrammingisimportantbecauseitistheonlyexactandecientmethodformaximizingutilityoverfuturetime.
Inthegeneralsituation,whererandomdisturbancesandnonlinearityareexpected,ADPisimportantbecauseitprovidesboththelearningcapabilityandthepossibilityofreducingcomputationalcosttoan7aordablelevel.
Forthisreason,ADPistheonlyapproachwehavetoimitatingthiskindofabilityofthebrain.
ThesimilaritybetweensomeADPdesignsandthecircuitryofthebrainhasbeendiscussedatlengthin[10]and[11].
Forexample,thereisanimportantstructureinthebraincalledthelimbicsystemwhichperformssomekindsofevaluationorreinforcementfunctions,verysimilartothefunctionsoftheneuralnetworksthatmustapproximatetheJfunctionofdynamicprogramming.
Thelargestpartofthelimbicsystem,calledthehippocampus,isknowntopossessahigherdegreeoflocalrecurrency[8].
Ingeneral,therearetwowaystomakeclassicalcontrollersstabledespitegreatuncertaintyaboutparametersoftheplanttobecontrolled.
Forexample,incontrollingahighspeedaircraft,thelocationofthecenterofthegravityisnotknown.
Thecenterofgravityisnotknownexactlybecauseitdependsonthecargooftheairplaneandthelocationofthepassengers.
Onewaytoaccountforsuchuncertaintiesistouseadaptivecontrolmethods.
Wecangetsimilarresults,butmoreassuranceofstabilityinmostcases[16]byusingrelatedneuralnetworkmethods,suchasadaptivecriticswithrecurrentnetworks.
Itislikeadaptivecontrolbutmoregeneral.
ThereisanotherapproachcalledrobustcontrolorH∞control,whichtrystodesignaxedcontrollerwhichremainsstableoveralargerangeinparameterspace.
BarasandPatel[31]haveforthersttimesolvedthegeneralproblemofH∞controlforgeneralpartiallyobservednonlinearplants.
Theyhaveshownthatthisproblemreducestoaprobleminnonlinear,stochasticoptimization.
Adaptivedynamicprogrammingmakesitpossibletosolvelargescaleproblemsofthistype.
2.
3ImportanceoftheSRNtoADPADPsystemsalreadyexistwhichperformrelativelysimplecontroltaskslikestabilizinganaircraftasitlandsunderwindyconditions[12].
Howeverthiskindoftaskdoesnotreallyrepresentthehighestlevelofintelligenceorplanning.
Trueintelligentcontrolrequirestheabilitytomakedecisionswhenfuturetimeperiodswillfollowacomplicated,unknownpathstartingfromtheinitialstate.
Oneexampleofachallengeforintelligentcontrolistheproblemofnavigatingamazewhichwewilldiscussinchapter4.
Atrueintelligentcontrolsystemshouldbeabletolearnthiskindoftask.
However,theADPsystemsinusetodaycouldneverlearnthiskindoftask.
TheyuseconventionalneuralnetworkstoapproximatetheJfunction.
BecausetheconventionalMLPcannotapproximatesuchaJfunction,wemaydeducethatADPsystemconstructedonlyfromMLPswillneverbeabletodisplaythiskindofintelligentcontrol.
Therefore,itisessentialthatwecanndakindofneuralnetworkwhichcanperformthiskindoftask.
Aswewillshow,theSRNcanllthiscrucialgap.
ThereareadditionalreasonsforbelievingthattheSRNmaybecrucialtointelligentcontrolasdiscussedinchapter13of[9].
83AlternativeFormsofRecurrentNetworks3.
1RecurrentNetworksinGeneralThereisahugeliteratureonrecurrentnetworks.
Biologistshaveusedmanyrecurrentmodelsbecausetheexistenceofrecurrencyinthebrainisobvious.
However,mostoftherecurrentnetworksimplementedsofarhavebeenclas-sicstylerecurrentnetworks,asshownonthelefthandofFigure2.
Mostofthesenetworksareformulatedfromordinarydierentialequation(ODE)sys-tems.
UsuallytheirlearningisbasedonarestrictedconceptofHebbianlearn-ing.
Originallyintheneuralnetworkeld,themostpopularneuralnetworkswererecurrentnetworkslikethosewhichHopeld[14]andGrossberg[15]usedtoprovideassociativememory.
FEATUREEXTRACTIONART,SOM,.
.
.
MINIMIZATIONHOPFIELD,CAUCHYCLASSICALRECURRENTNETWORKSRECURRENTNETWORKSTLRNSRN(DynamicSystems,Prediction)(Betterfunctionapproximation)ASSOCIATIVEMEMORYSTATICFUNCTIONCLUSTERINGHOPFIELD,HASSOUNFigure2:RecurrentnetworksAssociativememorynetworkscanactuallybeappliedtosupervisedlearning.
Butinactualitytheircapabilitiesareverysimilartothoseoflook-uptablesandradialbasisfunctions.
Theymakepredictionsbasedonsimilaritytoprevious9examplesorprototypes.
Theydonotreallytrytoestimategeneralfunctionalrelationships.
Asaresultthesemethodshavebecomeunpopularinpracticalapplicationsofsupervisedlearning.
ThetheoremsofBarrondiscussedintheIntroductionshowthatMLPsdoprovidebetterfunctionapproximationthandosimplemethodsbasedonsimilarity.
Therehasbeensubstantialprogressinthepastfewyearsindevelopingnewassociativememorydesigns.
Nevertheless,theMLPisstillbetterforthespecictaskoffunctionapproximationwhichisthefocusofthispaper.
Inasimilarway,classicrecurrentnetworkshavebeenusedfortaskslikeclustering,featureextractionandstaticfunctionoptimization.
Butthesearedierentproblemsfromwhatwearetryingtosolvehere.
Actuallytheproblemofstaticoptimizationwillbeconsideredinfuturestagesofthisresearch.
WehopethattheSRNcanbeusefulinsuchappli-cationsafterwehaveuseditforsupervisedlearning.
WhenpeopleusetheclassicHopeldnetworksforstaticoptimization,theyspecifyalltheweightsandconnectionsinadvance[14].
Thishaslimitedthesuccessofthiskindofnetworkforlargescaleproblemswhereitisdiculttoguesstheweights.
WiththeSRNwehavemethodstotraintheweightsinthatkindofstructure.
Thustheguessingisnolongerneeded.
However,touseSRNsinthatapplicationrequiresrenementbeyondthescopeofthispaper.
TherehavealsobeenresearchersusingODEneuralnetworkswhohavetriedtousetrainingschemesbasedonaminimizationoferrorinsteadofHebbianapproaches.
However,inpracticalapplicationsofsuchnetworks,itisimportanttoconsidertheclockratesofcomputationanddatasampling.
Forthatreason,itisbotheasierandbettertouseerrorminimizingdesignsbasedondiscretetimeratherthanODE.
3.
2StructureofDiscrete-TimeRecurrentNetworksIftheimportanceofneuralnetworksismeasuredbythenumberofwordspub-lished,thentheclassicnetworksdominatetheeldofrecurrentnetworks.
How-ever,ifthevalueismeasuredbasedoneconomicvalueofpracticalapplication,thentheeldisdominatedbytime-laggedrecurrentnetworks(TLRNs).
ThepurposeoftheTLRNistopredictorclassifytime-varyingsystemsusingre-currencyasawaytoprovidememoryofthepast.
TheSRNhassomerelationwiththeTLRNbutitisdesignedtoperformafundamentallydierenttask.
TheSRNusesrecurrencytorepresentmorecomplexrelationshipsbetweenoneinputvectorX(t)andoneoutputY(t)withoutconsiderationoftheothertimest.
Figure3andFigure4showusmoredetailsabouttheTLRNandtheSRN.
Incontrolapplications,u(t)representsthecontrolvariableswhichweusetocontroltheplant.
Forexample,ifwedesignacontrollerforacarengine,theX(t)variablesarethedatawegetfromoursensors.
Theu(t)variableswouldincludethevalvesettingswhichweusetotrytocontroltheprocessofcombustion.
TheR(t)variablesprovideawayfortheneuralnetworkstorememberpasttime10Z-1X(t)R(t)X(t+1)u(t)TLRNFigure3:Timelaggedrecurrentnetwork(TLRN)XfyFigure4:Simultaneousrecurrentnetwork(SRN)11cycles,andtoimplicitlyestimateimportantvariableswhichcannotbeobserveddirectly.
Infact,theapplicationofTLRNstoautomobilecontrolisthemostvaluableapplicationofrecurrentnetworkseverdevelopedsofar.
Asimultaneousrecurrentnetwork(Figure4)isdenedasamapping:Y(t)=F(X(t),W)(3)whichiscomputedbyiteratingoverthefollowingequation:y(n+1)(t)=f(y(n)(t),X(t),W)(4)wherefissomesortoffeed-forwardnetworkorsystem,andYisdenedas:Y(t)=limn→∞y(n)(t)(5)WhenweuseYinthispaper,weusen=20insteadof∞here.
InFigure4,theoutputsoftheneuralnetworkcomebackagainasinputstothesamenetwork.
However,inconceptthereisnotimedelay.
Theinputsandoutputsshouldbesimultaneous.
Thatiswhyitiscalledasimultaneousrecurrentnetwork(SRN).
Inpractice,ofcourse,therewillalwaysbesomephys-icaltimedelaybetweentheoutputsandtheinputs.
HoweveriftheSRNisimplementedinfastcomputers,thistimedelaymaybeverysmallcomparedtothedelaybetweendierentframesofinputdata.
InFigure4,Xreferstotheinputdataatthecurrenttimeframet.
Thevectoryrepresentsthetemporaryoutputofthenetwork,whichisthenrecycledasanadditionalsetofinputstothenetwork.
AtthecenteroftheSRNisactuallythefeed-forwardnetworkwhichimplementsthefunctionf.
(IndesigninganSRN,youcanchooseanyfeed-forwardnetworkorsystemasyoulike.
Thefunctionfsimplydescribeswhichnetworkyouuse).
TheoutputoftheSRNatanytimetissimplythelimitofthetemporaryoutputy.
InEquation(3)and(4),noticethattherearetwointegers—nandt—whichcouldbothrepresentsomekindoftime.
Theintegertrepresentsaslowerkindoftimecycle,likethedelaybetweenframesofincomingdata.
Theintegernrepresentsafasterkindoftime,likethecomputingcycleofafastelectronicchip.
Forexample,ifwebuildacomputertoanalyzeimagescomingfromamoviecamera,"t"and"t+1"representtwosuccessiveincomingpictureswithamoviecamera.
Thereareusuallyonly32framespersecond.
(Inthehumanbrain,itseemsthatthereareonlyabout10framespersecondcomingintotheneocortex.
)Butifweuseafastneuralnetworkchip,thecomputationalcycle—thetimebetween"n"and"n+1"—couldbeassmallasamicrosecond.
Inactuality,itisnotnecessarytochoosebetweentime-laggedrecurrency(fromttot+1)andsimultaneousrecurrency(fromnton+1).
Itispossibletobuildahybridsystemwhichcontainsbothtypesofrecurrency.
Thiscouldbeveryusefulinanalyzingdatalikemoviepictures,whereweneedbothmemoryandsomeabilitytosegmenttheimages.
[9]discusseshowtobuildsuchahybrid.
12However,beforebuildingsuchahybrid,wemustrstlearntomakeSRNsworkbythemselves.
Finally,pleasenotethattheTLRNisnottheonlykindofneuralnetworkusedinpredictingdynamicalsystems.
Evenmorepopularisthetimedelayedneuralnetwork(TDNN),showninFigure5.
TheTDNNispopularbecauseitiseasytouse.
However,ithaslesscapability,inprinciple,becauseithasnoabilitytoestimateunknownvariables.
Itisespeciallyweakwhensomeofthesevariableschangeslowlyovertimeandrequirememorywhichpersistsoverlongtimeperiods.
Inaddition,theTLRNtstherequirementsofADPdirectly,whiletheTDNNdoesnot[9][16].
X(t)X(t-1)X(t-k)TDNNX(t+1)u(t-k)u(t-1)u(t)Figure5:Timedelayedneuralnetwork(TDNN)3.
3TrainingofSRNsandTLRNsTherearemanytypesoftrainingthathavebeenusedforrecurrentnetworks.
Dierenttypesoftraininggiverisetodierentkindsofcapabilitiesfordierenttasks.
ForthetaskswhichwehavedescribedfortheSRNandtheTLRN,theproperformsoftrainingallinvolvesomecalculationofthederivativesoferrorwithrespectstotheweights.
Usuallyafterthesederivativesareknown,theweightsareadaptedaccordingtoasimpleformulaasfollows:newWi,j=oldWi,jLRErrorWi,j(6)whereLRiscalledthelearningrate.
TherearevemainwaystotrainSRNs,allbasedondierentmethodsforcalculatingorapproximatingthederivatives.
FourofthesemethodscanalsobeusedwithTLRNs.
Somecanbeusedforcontrolapplications.
Butthedetailsofthoseapplicationsarebeyondthescopeofthispaper.
Theseve13typesoftrainingarelistedinFigure6.
Forthispaper,wehaveusedtwoofthesemethods:Backpropagationthroughtime(BTT)andTruncation.
TypesofSRNSimultaneousBackpropagationBackpropagationThroughTimeForwardPropagationTrainingTruncationErrorCriticsFigure6:TypesofSRNTrainingThevemethodsare:1.
Backpropagationthroughtime(BTT).
Thismethodandforwardpropa-gationarethetwomethodswhichcalculatethederivativesexactly.
BTTisalsolessexpensivethanforwardpropagation.
2.
Truncation.
Thisisthesimplestandleastexpensivemethod.
Itusesonlyonesimplepassofbackpropagationthroughthelastiterationofthemodel.
TruncationisprobablythemostpopularmethodusedtoadaptSRNseventhoughthepeoplewhouseitmostlyjustcallitordinarybackpropagation.
3.
Simultaneousbackpropagation.
Thisismorecomplexthantruncation,butitstillcanbeusedinrealtimelearning.
Itcalculatesderivativeswhichareexactintheneighborhoodofequilibriumbutitdoesnotaccountforthedetailsofthenetworkbeforeitreachestheneighborhoodofequilibrium.
4.
Errorcritics(showninFigure7).
ThisprovidesageneralapproximationtoBTTwhichissuitableforuseinreal-timelearning[9].
14TLRNTLRNCriticErrorCriticErrorErrorErrorR(t)X(t)u(t)X(t)u(t)X(t+1)X(t+1)X(t=1)u(t+1)R(t+1)λ^(t)λ^λ(t)(t+1)Figure7:ErrorCritics155.
Forwardpropagation.
This,likeBTT,calculatesexactderivatives.
Itisoftenconsideredsuitableforreal-timelearningbecausethecalculationsgoforwardintime.
However,whentherearenneuronsandmconnections,thecostofthismethodperunitoftimeisproportionaltonm.
Becauseofthishighcost,forwardpropagationisnotreallybrain-likeanymorethanBTT.
3.
3.
1Backpropagationthroughtime(BTT)BTTisageneralmethodforcalculatingallthederivativeofanyoutcomeorresultofaprocesswhichinvolvesrepeatedcallstothesamenetworkornet-worksusedtohelpcalculatesomekindofnaloutcomevariableorresultE.
Insomeapplications,Ecouldrepresentutility,performance,costorothersuchvariables.
Butinthispaper,Ewillbeusedtorepresenterror.
BTTwasrstproposedandimplementedin[17].
ThegeneralformofBTTisasfollows:fork=1toTdoforwardcalculation(k);calculateresultE;calculatedirectderivativesofEwithrespecttooutputsofforwardcalculations;fork=Tto1backpropagatethroughforwardscalculation(k),calculatingrun-ningtotalswhereappropriate.
ThesestepsareillustratedinFigure8.
Noticethatthisalgorithmcanbeappliedtoallkindsofcalculations.
ThuswecanapplyittocaseswherekrepresentsdataframestasintheTLRNs,ortocaseswherekrepresentsinternaliterationsnasintheSRNs.
AlsonotethateachboxofcalculationreceivesinputfromsomedashedlineswhichrepresentthederivativesofEwithrespecttotheoutputofthebox.
Inordertocalculatethederivativescomingoutofeachcalculationbox,onesimplyusesbackpropagationthroughthecalculationofthatboxstartingoutfromtheincomingderivatives.
WewillexplaininmoredetailhowthisworksintheSRNcaseandtheTLRNcase.
SofarasweknowBTThasbeenappliedinpublishedworkingsystemsforTLRNsandforcontrol,butnotyetforSRNsuntilnow.
However,Rumelhart,HintonandWilliams[18]didsuggestthatsomeoneshouldtrythis.
TheapplicationofBTTforTLRNsisdescribedatlengthin[2]and[9].
TheprocedureisillustratedinFigure9.
Inthisexamplethetotalerrorisactuallythesumoferrorovereachtimetwheretgoesfrom1toT.
ThereforetheoutputsoftheTLRNateachtimet(t#include#include#includevoidF_NET2(doubleF_Yhat,doubleW[30][30],doublex[30],intn,intm,intN,doubleF_W[30][30],doubleF_net[30],doubleF_Ws[30],doubleWs,doubleF_x[30]);voidNET(doubleW[30][30],doublex[30],doubleww[30],intn,intm,intN,doubleYhat[30]);41voidsynthesis(intB[30][30],intA[30][30],intn1,intn2);voidpweight(doubleWs,doubleF_Ws_T,doubleww[30],doubleF_net_T[30],doubleW[30][30],doubleF_W_T[30][30],intn,intN,intm);intminimum(ints,intt,intu,intv);intmin(intk,intl);doublef(doublex);voidmain(){inti,j,it,iz,ix,iy,lt,m,maxt,n,n1,n2,nn,nm,N,p,q,po,t,TT;intA[30][30],B[30][30];doublea,b,dot,e,e1,e2,es,mu,s,sum,F_Ws_T,Ws,F_Yhat,wi;doubleW[30][30],x[30],F_net_T[30],F_Ws[30],F_W_O[30][30],F_W[30][30];doubleF_W_T[30][30],F_net[30],ww[30],yy[21][12][8][8];doubleYhat[30],F_y[21][12][8][8],F_x[30],F_Jhat[30][30];doubleS_F_W1,S_F_W2,Lr_W,S_F_net1,S_F_net2,Lr_ww,Lr_Ws,F_Ws_O;doubley[50][50],F_net_O[50],F_Ws1,F_Ws2,W_O[50][50],ww_O[50],Ws_O;FILE*f;/*Numberofinputs,neuronsandoutput:7,3,1*//*'n'isthenumberoftheactiveneurons*//*'m'and'N'botharethenumberofinputs*//*'nm'isthenumberofmemoryis:5*//*'nn+1'*'nn+1'isthesizeofthemaze'*//*'TT'isthenumberoftrials*//*'lt'isthenumberoftheintervaltime*//*'maxt'isthemaxnumberforTinfigure[8]*//*Lr-Ws,Lr_wwandLr_WarethelearningratesforWs,wwandW*/a=0.
9;b=0.
2;n=5;m=11;N=11;nn=6;nm=5;TT=30000;lt=50;maxt=20;wi=25;Ws=40;e=0;po=pow(2,31)-1;/*InitialvaluesofOld*/F_Ws_O=1;for(i=m+1;i-1;q--){for(ix=0;ixS_F_W2)||(S_F_W1==S_F_W2))Lr_W=Lr_W*(a+b);elseif(S_F_W1S_F_net2)||(S_F_net1==S_F_net2))Lr_ww=Lr_ww*(a+b);elseif(S_F_net1F_Ws2)||(F_Ws1==F_Ws2))Lr_Ws=Lr_Ws*(a+b);elseif(F_Ws1m;i--){for(j=i+1;j0;i--)for(j=m+1;jl)r=l;elser=k;returnr;}voidpweight(doubleWs,doubleF_Ws_T,doubleww[30],doubleF_net_T[30],doubleW[30][30],doubleF_W_T[30][30],intn,intN,intm){inti,j;for(i=m+1;i
WerbosAbstractThispaperwillshowthatanewneuralnetworkdesigncansolveanexampleofdicultfunctionapproximationproblemswhicharecrucialtotheeldofapproximatedynamicprogramming(ADP).
Althoughconven-tionalneuralnetworkshavebeenproventoapproximatesmoothfunctionsverywell,theuseofADPforproblemsofintelligentcontrolorplanningrequirestheapproximationoffunctionswhicharenotsosmooth.
Asanexample,thispaperstudiestheproblemofapproximatingtheJfunctionofdynamicprogrammingappliedtothetaskofnavigatingmazesingen-eralwithouttheneedtolearneachindividualmaze.
Conventionalneuralnetworks,likemulti-layerperceptrons(MLPs),cannotlearnthistask.
Butanewtypeofneuralnetworks,simultaneousrecurrentnetworks(SRNs),candosoasdemonstratedbysuccessfulinitialtests.
Thepaperalsoex-aminestheabilityofrecurrentneuralnetworkstoapproximateMLPsandviceversa.
Keywords:Simultaneousrecurrentnetworks(SRNs),multi-layerpercep-trons(MLPs),approximatedynamicprogramming,mazenavigation,neuralnet-works.
1Introduction1.
1PurposeThispaperhasthreegoals:First,todemonstratethevalueofanewclassofneuralnetworkwhichpro-videsacrucialcomponentneededforbrain-likeintelligentcontrolsystemsforthefuture.
Second,todemonstratethatthisnewkindofneuralnetworkprovidesbetterfunctionapproximateabilityforuseinmoreordinarykindsofneuralnetworkapplicationsforsupervisedlearning.
Third,todemonstratesomepracticalimplementationtechniquesnecessarytomakethiskindofnetworkactuallyworkinpractice.
X(t)Y(t)SupervisedLearningSystemActualY(t)^Figure1:Whatissupervisedlearning1.
2BackgroundAtpresent,intheneuralnetworkeldperhaps90%ofneuralnetworkapplica-tionsinvolvetheuseofneuralnetworksdesignedtoperformanceataskcalledsupervisedlearning(Figure1).
Supervisedlearningisthetaskoflearninganon-linearfunctionwhichmayhaveseveralinputsandseveraloutputsbasedonsomeexamplesofthefunction.
Forexample,incharacterrecognition,theinputsmaybeanarrayofpixelsseenfromacamera.
Thedesiredoutputsofthenetworkmaybeaclassicationofcharacterbeingseen.
Anotherexamplewouldbeforintelligentsensinginthechemicalindustrywheretheinputsmightbespectraldatafromobservingabatchofchemicals,andthedesiredoutputswouldbetheconcentrationsofthedierentchemicalsinthebatch.
Thepurposeofthisapplicationistopredictorestimatewhatisinthebatchwithouttheneedforexpensiveanalyticaltests.
Theworkinthispaperwillfocustotallyoncertaintasksinsupervisedlearning.
Eventhoughexistingneuralnetworkscanbeusedinsupervisedlearn-ing,therecanbeperformanceproblemsdependingonwhatkindoffunctionislearned.
Manypeoplehaveprovenmanytheoremstoshowthatneuralnetworks,fuzzylogic,Taylortheoriesandotherfunctionapproximationhaveauniversalabilitytoapproximatefunctionsontheconditionthatthefunctionshavecertainpropertiesandthatthereisnolimitonthecomplexityoftheapproximation.
Inpractice,manyapproximationschemesbecomeuselesswhentherearemanyinputvariablesbecausetherequiredcomplexitygrowsatanexponentialrate.
Forexample,onewaytoapproximateafunctionwouldbetoconstructatableofthevaluesofthefunctionatcertainpointsinthespaceofpossibleinputs.
Supposethereare30inputvariablesandweconsider10possiblevaluesofeachinput.
Inthatcase,thetablemusthave1030numbersinit.
Thisisnotusefulinpracticeformanyreasons.
Actually,however,manypopular2approximationmethodslikeradialbasisfunction(RBF)aresimilarinspirittoatableofvalues.
Intheeldofsupervisedlearning,AndrewBarron[30]hasprovedsomefunc-tionapproximationtheoremswhicharemuchmoreusefulinpractice.
Hehasproventhatthemostpopularformofneuralnetworks,themulti-layerpercep-tron(MLP),canapproximateanysmoothfunction.
Unlikethecasewiththelinearbasisfunctions(likeRBFandTaylorseries),thecomplexityofthenet-workdoesnotgrowrapidlyasthenumberofinputvariablesgrows.
Unfortunatelytherearemanypracticalapplicationswherethefunctionstobeapproximatedarenotsmooth.
Insomecases,itisgoodenoughjusttoaddextralayerstoanMLP[1]ortouseageneralizedMLP[2].
However,therearesomedicultproblemswhichariseineldslikeintelligentcontrolorimageprocessingorevenstochasticsearchwherefeed-forwardnetworksdonotappearpowerfulenough.
1.
3SummaryandOrganizationofThisPaperThemaingoalofthispaperistodemonstratethecapabilityofadierentkindofsupervisedlearningsystembasedonakindofrecurrentnetworkcalledsimultaneousrecurrentnetwork(SRN).
Inthenextchapterwewillexplainwhythiskindofimprovedsupervisedlearningsystemwillbeveryimportanttointelligentcontrolandtoapproximatedynamicprogramming.
Ineectthisworkonsupervisedlearningistherststepinamulti-stepeorttobuildmorebrain-likeintelligentsystems.
ThenextstepwouldbetoapplytheSRNtostaticoptimizationproblems,andthentointegratetheSRNsintolargesystemsforADP.
Eventhoughintelligentcontrolisthemainmotivationforthiswork,theworkmaybeusefulforotherareasaswell.
Forexample,inzipcoderecog-nition,AT&T[3]hasdemonstratedthatfeed-forwardnetworkscanachieveahighlevelofaccuracyinclassifyingindividualdigits.
However,AT&Tandtheothersstillhavedicultyinsegmentingthetotalzipcodesintoindividualdig-its.
ResearchonhumanvisionbyvonderMalsburg[4]andothershassuggestedthatsomekindsofrecurrencyinneuralnetworksarecrucialtotheirabilitiesinimagesegmentationandbinocularvision.
Furthermore,researchersinimageprocessinglikeLaveenKanalhaveshowedthatiterativerelaxationalgorithmsarenecessaryeventoachievemoderatesuccessinsuchimageprocessingtasks.
ConceptuallytheSRNcanlearnanoptimaliterativealgorithm,buttheMLPcannotrepresentanyiterativealgorithms.
Insummary,thoughwearemostinterestedinbrain-likeintelligentcontrol,thedevelopmentofSRNscouldleadtoveryimportantapplicationsinareassuchasimageprocessinginthefuture.
Thenetworkdescribedinthispaperisuniqueinseveralrespects.
However,itiscertainlynottherstserioususeofarecurrentneuralnetwork.
Chapter3ofthispaperwilldescribetheexistingliteratureonrecurrentnetworks.
Itwilldescribetherelationshipbetweenthisnewdesignandotherdesignsinthe3literature.
Roughlyspeaking,thevastbulkofresearchinrecurrentnetworkshasbeenacademicresearchusingdesignsbasedonordinarydierentialequa-tions(ODE)toperformsometasksverydierentfromsupervisedlearning—taskslikeclustering,associativememoryandfeatureextraction.
ThesimpleHebbianlearningmethods[13]usedforthosetasksdonotleadtothebestper-formanceinsupervisedlearning.
Manyengineershaveusedanothertypeofrecurrentnetwork,thetimelaggedrecurrentnetwork(TLRN),wheretherecur-rencyisusedtoprovidememoryofpasttimeperiodsforuseinforecastingthefuture.
However,thatkindofrecurrencycannotprovidetheiterativeanalysiscapabilitymentionedabove.
VeryfewresearchershavewrittenaboutSRNs,atypeofrecurrentnetworkdesignedtominimizeerrorandlearnanoptimaliterativeapproximationtoafunction.
ThisiscertainlytherstuseofSRNstolearnaJfunctionfromdynamicprogrammingwhichwillbeexplainedmoreinchapter2.
ThismayalsobetherstempiricaldemonstrationoftheneedforadvancedtrainingmethodstopermitSRNstolearndicultfunctions.
Chapter4willexplaininmoredetailthetwotestproblemswehaveusedfortheSRNandtheMLP,aswellasthedetailsofarchitectureandlearningprocedure.
ThersttestproblemwasusedmainlyasaninitialtestofasimpleformofSRNs.
Inthisproblem,wetriedtotestthehypothesisthatanSRNcanalwayslearntoapproximatearandomlychosenMLP,butnotviceversa.
Althoughourresultsareconsistentwiththathypothesis,thereisroomformoreextensiveworkinthefuture,suchasexperimentswithdierentsizesofneuralnetworksandmorecomplexstatisticalanalysis.
ThemaintestprobleminthisworkwastheproblemoflearningtheJfunctionofdynamicprogramming.
Foramazenavigationproblem,manyneuralnetworkresearchershavewrittenaboutneuralnetworkswhichlearnanoptimalpolicyofactionforoneparticularmaze[5].
ThispaperwilladdressthemoredicultproblemoftraininganeuralnetworktoinputapictureofamazeandoutputtheJfunctionforthismaze.
WhentheJfunctionisknown,itisatriviallocalcalculationtondthebestdirectionofmovement.
Thiskindofneuralnetworkshouldnotrequireretrainingwheneveranewmazeisencountered.
Insteaditshouldbeabletolookatthemazeandimmediately"see"theoptimalstrategy.
Trainingsuchanetworkisaverydicultproblemwhichhasneverbeensolvedinthepastwithanykindofneuralnetwork.
Alsoitistypicalofthechallengesoneencountersintrueintelligentcontrolandplanning.
Thispaperhasdemonstratedaworkingsolutiontothisproblemforthersttime.
Nowthatasystemisworkingonaverysimpleformforthisproblem,itwouldbepossibleinthefuturetoperformmanytestsoftheabilityofthissystemtogeneralizeitssuccesstomanymazes.
Inordertosolvethemazeproblem,itwasnotsucientonlytouseanSRN.
TherearemanychoicestomakewhenimplementingthegeneralideaofSRNsorMLPs.
Chapter5willdescribeindetailhowthesechoicesweremadeinthiswork.
Themostimportantchoiceswere:41.
BothfortheMLPandforthefeed-forwardcoreoftheSRNweusedthegeneralizedMLPdesign[2]whicheliminatestheneedtodecideonthenumberoflayers.
2.
Forthemazeproblem,weusedacellularorweight-sharingarchitecturewhichexploitsthespatialsymmetryoftheproblemandreducesdramaticallythenumberofweights.
Ineectwesolvedthemazeproblemusingonlyvedistinctneurons.
Thereareinterestingparallelsbetweenthisnetworkandthehippocampusofthehumanbrain.
3.
Forthemazeproblem,anadaptivelearningrate(ALR)procedurewasusedtopreventoscillationandensureconvergence.
4.
InitialvaluesfortheweightsandtheinitialinputvectorfortheSRNwerechosenessentiallyatrandom,byhand.
Inthefuture,moresystematicmethodsareavailable.
Butthiswassucientforsuccessinthiscase.
FinallyChapter6willdiscussthesimulationresultsinmoredetail,givetheconclusionsofthispaperandmentionsomepossibilitiesforfuturework.
2MotivationInthischapterwewillexplaintheimportanceofthiswork.
Asdiscussedabove,thepapershowshowtouseanewtypeofneuralnetworkinordertoachievebetterfunctionapproximationthanwhatisavailablefromthetypesofneu-ralnetworkswhicharepopulartoday.
Thischapterwilltrytoexplainwhybetterfunctionapproximationisimportanttoapproximatedynamicprogram-ming(ADP),intelligentcontrolandunderstandingthebrain.
ImageprocessingandotherapplicationshavealreadybeendiscussedintheIntroduction.
Thesethreetopics—ADP,intelligentcontrolandunderstandingthebrain—areallcloselyrelatedtoeachotherandprovidetheoriginalmotivationfortheworkofthispaper.
Thepurposeofthispaperistomakeacorecontributiontodevelopingthemostpowerfulpossiblesystemforintelligentcontrol.
Inordertobuildthebestintelligentcontrolsystems,weneedtocombinethemostsuitablemathematicstogetherwithsomeunderstandingofnaturalintelligenceinthebrain.
Thereisalotofinterestinintelligentcontrolintheworld.
Somecontrolsystemswhicharecalledintelligentareactuallyveryquickandeasythings.
Therearemanypeoplewhotrytomovestepbysteptoaddintelligenceintocontrol,butastep-by-stepapproachmaynotbeenoughbyitself.
Sometimestoachieveacomplexdicultgoal,itisnecessarytohaveaplan,thussomepartsoftheintelligentcontrolcommunityhavedevelopedamoresystematicvisionorplanforhowitcouldbepossibletoachieverealintelligentcontrol.
First,onemustthinkaboutthequestionofwhatisintelligentcontrol.
Then,insteadoftryingtoanswerthisquestioninonestep,wetrytodevelopaplantoreachthedesign.
Actuallytherearetwoquestions:51.
Howcouldwebuildanarticialsystemwhichreplicatesthemaincapa-bilitiesofbrain-likeintelligence,somehowuniedtogetherastheyareuniedtogetherinthebrain2.
Howcanweunderstandwhatarethecapabilitiesinthebrainandhowtheyareorganizedinafunctionalengineeringviewi.
e.
howarethosecircuitsinthehumanbrainarrangedtolearnhowtoperformdierenttasksItwouldbebesttounderstandhowthehumanbrainworksbeforebuildinganarticialsystem.
However,atthepresenttime,ourunderstandingofthebrainislimited.
Butatleastweknowthatlocalrecurrencyplayscriticalruleinthehigherpartofthehumanbrain[6][7][8][4].
AnotherreasontouseSRNsisthatSRNscanbeveryusefulinADPmath-ematically.
NowwewilldiscusswhatADPcandoforintelligentcontrolandunderstandingthebrain.
Theremainderofthischapterwilladdressthreequestionsinorder:1.
WhatisADP2.
WhatistheimportanceofADPtointelligentcontrolandunderstandingthebrain3.
WhatistheimportanceofSRNstoADP2.
1WhatisADPandJFunctionToexplainwhatisADP,letusconsidertheoriginalBellmanequation[9]:J(R(t))=maxu(t)(U(R(t),u(t))+)/(1+r)U0(1)whererandu0areconstantsthatareusedonlyininnite-time-horizonproblemsandthenonlysometimes,andwheretheanglebracketsrefertoexpectationvalue.
Inthispaperweactuallyuse:J(R(t))=maxu(t)(U(R(t),u(t))+)(2)sincethemazeproblemdonotinvolveaninnitetime-horizon.
InsteadofsolvingforthevalueofJineverypossiblestate,R(t),wecanuseafunctionapproximationmethodlikeneuralnetworkstoapproximatetheJfunction.
Thisiscalledapproximatedynamicprogramming(ADP).
ThispaperisnotdoingtrueADPbecauseintrueADPwedonotknowwhattheJfunctionisandmustthereforeuseindirectmethodstoapproximateit.
However,beforewetrytouseSRNsasacomponentofanADPsystem,itmakessensetorsttesttheabilityofanSRNtoapproximateaJfunction,inprinciple.
NowwewilltrytoexplainwhatistheintuitivemeaningoftheBellmanequation(Equation(1))andtheJfunctionaccordingtothetreatmenttakenfrom[2].
TounderstandADP,onemustrstreviewthebasicsofclassicaldynamicprogramming,especiallytheversionsdevelopedbyHoward[28]andBertsekas.
6Classicaldynamicprogrammingistheonlyexactandecientmethodtocom-putetheoptimalcontrolpolicyovertime,inageneralnonlinearstochasticenvi-ronment.
Theonlyreasontoapproximateitistoreducecomputationalcost,soastomakethemethodaordable(feasible)acrossawiderangeofapplications.
Indynamicprogramming,theusersuppliesautilityfunctionwhichmaytaketheformU(R(t),u(t))—wherethevectorRisaRepresentationorestimateofthestateoftheenvironment(i.
e.
thestatevector)—andastochasticmodeloftheplantorenvironment.
Then"dynamicprogramming"(i.
e.
solutionoftheBellmanequation)givesusbackasecondaryorstrategicutilityfunctionJ(R).
ThebasictheoremisthatmaximizingU(R(t),u(t))+J(R(t+1))yieldstheoptimalstrategy,thepolicywhichwillmaximizetheexpectedvalueofUaddedupoverallfuturetime.
Thusdynamicprogrammingconvertsadicultprob-leminoptimizingovermanytimeintervalsintoastraightforwardprobleminshort-termmaximization.
Inclassicaldynamicprogramming,wendtheexactfunctionJwhichexactlysolvestheBellmanequation.
InADP,welearnakindof"model"ofthefunctionJ;this"model"iscalleda"Critic.
"(Alternatively,somemethodslearnamodelofthederivativesofJwithrespecttothevariablesRi;thesecorrespondtoLagrangemultipliers,λi,andtothe"pricevariables"ofmicroeconomictheory.
SomemethodslearnafunctionrelatedtoJ,asintheAction-DependentAdaptiveCritic(ADAC)[29].
2.
2IntelligentControlandRobustControlTounderstandthehumanbrainscientically,wemusthavesomesuitablemath-ematicalconcepts.
Sincethehumanbrainmakesdecisionslikeacontrolsystem,itisanexampleofanintelligentcontrolsystem.
Neuroscientistsdonotyetun-derstandthegeneralabilityofthehumanbraintolearntoperformnewtasksandsolvenewproblemseventhoughtheyhavestudiedthebrainfordecades.
Somepeoplecomparethepastresearchinthiseldtowhatwouldhappenifwespentyearstostudyradioswithoutknowingthemathematicsofsignalprocessing.
Werstneedsomemathematicalideasofhowitispossibleforacomputingsystemtohavethiskindofcapabilitybasedondistributedparallelcomputation.
Thenwemustaskwhatarethemostimportantabilitiesofthehumanbrainwhichunifyallofitsmorespecicabilitiesinspecictasks.
Itwouldbeseenthatthemostimportantabilityofbrainistheabilitytolearnovertimehowtomakebetterdecisionsinordertobettermaximizethegoalsoftheorganism.
Thenaturalwaytoimitatethiscapabilityinengineeringsystemsistobuildsystemswhichlearnovertimehowtomakedecisionswhichmaximizesomemeasureofsuccessorutilityoverfuturetime.
Inthiscontext,dynamicprogrammingisimportantbecauseitistheonlyexactandecientmethodformaximizingutilityoverfuturetime.
Inthegeneralsituation,whererandomdisturbancesandnonlinearityareexpected,ADPisimportantbecauseitprovidesboththelearningcapabilityandthepossibilityofreducingcomputationalcosttoan7aordablelevel.
Forthisreason,ADPistheonlyapproachwehavetoimitatingthiskindofabilityofthebrain.
ThesimilaritybetweensomeADPdesignsandthecircuitryofthebrainhasbeendiscussedatlengthin[10]and[11].
Forexample,thereisanimportantstructureinthebraincalledthelimbicsystemwhichperformssomekindsofevaluationorreinforcementfunctions,verysimilartothefunctionsoftheneuralnetworksthatmustapproximatetheJfunctionofdynamicprogramming.
Thelargestpartofthelimbicsystem,calledthehippocampus,isknowntopossessahigherdegreeoflocalrecurrency[8].
Ingeneral,therearetwowaystomakeclassicalcontrollersstabledespitegreatuncertaintyaboutparametersoftheplanttobecontrolled.
Forexample,incontrollingahighspeedaircraft,thelocationofthecenterofthegravityisnotknown.
Thecenterofgravityisnotknownexactlybecauseitdependsonthecargooftheairplaneandthelocationofthepassengers.
Onewaytoaccountforsuchuncertaintiesistouseadaptivecontrolmethods.
Wecangetsimilarresults,butmoreassuranceofstabilityinmostcases[16]byusingrelatedneuralnetworkmethods,suchasadaptivecriticswithrecurrentnetworks.
Itislikeadaptivecontrolbutmoregeneral.
ThereisanotherapproachcalledrobustcontrolorH∞control,whichtrystodesignaxedcontrollerwhichremainsstableoveralargerangeinparameterspace.
BarasandPatel[31]haveforthersttimesolvedthegeneralproblemofH∞controlforgeneralpartiallyobservednonlinearplants.
Theyhaveshownthatthisproblemreducestoaprobleminnonlinear,stochasticoptimization.
Adaptivedynamicprogrammingmakesitpossibletosolvelargescaleproblemsofthistype.
2.
3ImportanceoftheSRNtoADPADPsystemsalreadyexistwhichperformrelativelysimplecontroltaskslikestabilizinganaircraftasitlandsunderwindyconditions[12].
Howeverthiskindoftaskdoesnotreallyrepresentthehighestlevelofintelligenceorplanning.
Trueintelligentcontrolrequirestheabilitytomakedecisionswhenfuturetimeperiodswillfollowacomplicated,unknownpathstartingfromtheinitialstate.
Oneexampleofachallengeforintelligentcontrolistheproblemofnavigatingamazewhichwewilldiscussinchapter4.
Atrueintelligentcontrolsystemshouldbeabletolearnthiskindoftask.
However,theADPsystemsinusetodaycouldneverlearnthiskindoftask.
TheyuseconventionalneuralnetworkstoapproximatetheJfunction.
BecausetheconventionalMLPcannotapproximatesuchaJfunction,wemaydeducethatADPsystemconstructedonlyfromMLPswillneverbeabletodisplaythiskindofintelligentcontrol.
Therefore,itisessentialthatwecanndakindofneuralnetworkwhichcanperformthiskindoftask.
Aswewillshow,theSRNcanllthiscrucialgap.
ThereareadditionalreasonsforbelievingthattheSRNmaybecrucialtointelligentcontrolasdiscussedinchapter13of[9].
83AlternativeFormsofRecurrentNetworks3.
1RecurrentNetworksinGeneralThereisahugeliteratureonrecurrentnetworks.
Biologistshaveusedmanyrecurrentmodelsbecausetheexistenceofrecurrencyinthebrainisobvious.
However,mostoftherecurrentnetworksimplementedsofarhavebeenclas-sicstylerecurrentnetworks,asshownonthelefthandofFigure2.
Mostofthesenetworksareformulatedfromordinarydierentialequation(ODE)sys-tems.
UsuallytheirlearningisbasedonarestrictedconceptofHebbianlearn-ing.
Originallyintheneuralnetworkeld,themostpopularneuralnetworkswererecurrentnetworkslikethosewhichHopeld[14]andGrossberg[15]usedtoprovideassociativememory.
FEATUREEXTRACTIONART,SOM,.
.
.
MINIMIZATIONHOPFIELD,CAUCHYCLASSICALRECURRENTNETWORKSRECURRENTNETWORKSTLRNSRN(DynamicSystems,Prediction)(Betterfunctionapproximation)ASSOCIATIVEMEMORYSTATICFUNCTIONCLUSTERINGHOPFIELD,HASSOUNFigure2:RecurrentnetworksAssociativememorynetworkscanactuallybeappliedtosupervisedlearning.
Butinactualitytheircapabilitiesareverysimilartothoseoflook-uptablesandradialbasisfunctions.
Theymakepredictionsbasedonsimilaritytoprevious9examplesorprototypes.
Theydonotreallytrytoestimategeneralfunctionalrelationships.
Asaresultthesemethodshavebecomeunpopularinpracticalapplicationsofsupervisedlearning.
ThetheoremsofBarrondiscussedintheIntroductionshowthatMLPsdoprovidebetterfunctionapproximationthandosimplemethodsbasedonsimilarity.
Therehasbeensubstantialprogressinthepastfewyearsindevelopingnewassociativememorydesigns.
Nevertheless,theMLPisstillbetterforthespecictaskoffunctionapproximationwhichisthefocusofthispaper.
Inasimilarway,classicrecurrentnetworkshavebeenusedfortaskslikeclustering,featureextractionandstaticfunctionoptimization.
Butthesearedierentproblemsfromwhatwearetryingtosolvehere.
Actuallytheproblemofstaticoptimizationwillbeconsideredinfuturestagesofthisresearch.
WehopethattheSRNcanbeusefulinsuchappli-cationsafterwehaveuseditforsupervisedlearning.
WhenpeopleusetheclassicHopeldnetworksforstaticoptimization,theyspecifyalltheweightsandconnectionsinadvance[14].
Thishaslimitedthesuccessofthiskindofnetworkforlargescaleproblemswhereitisdiculttoguesstheweights.
WiththeSRNwehavemethodstotraintheweightsinthatkindofstructure.
Thustheguessingisnolongerneeded.
However,touseSRNsinthatapplicationrequiresrenementbeyondthescopeofthispaper.
TherehavealsobeenresearchersusingODEneuralnetworkswhohavetriedtousetrainingschemesbasedonaminimizationoferrorinsteadofHebbianapproaches.
However,inpracticalapplicationsofsuchnetworks,itisimportanttoconsidertheclockratesofcomputationanddatasampling.
Forthatreason,itisbotheasierandbettertouseerrorminimizingdesignsbasedondiscretetimeratherthanODE.
3.
2StructureofDiscrete-TimeRecurrentNetworksIftheimportanceofneuralnetworksismeasuredbythenumberofwordspub-lished,thentheclassicnetworksdominatetheeldofrecurrentnetworks.
How-ever,ifthevalueismeasuredbasedoneconomicvalueofpracticalapplication,thentheeldisdominatedbytime-laggedrecurrentnetworks(TLRNs).
ThepurposeoftheTLRNistopredictorclassifytime-varyingsystemsusingre-currencyasawaytoprovidememoryofthepast.
TheSRNhassomerelationwiththeTLRNbutitisdesignedtoperformafundamentallydierenttask.
TheSRNusesrecurrencytorepresentmorecomplexrelationshipsbetweenoneinputvectorX(t)andoneoutputY(t)withoutconsiderationoftheothertimest.
Figure3andFigure4showusmoredetailsabouttheTLRNandtheSRN.
Incontrolapplications,u(t)representsthecontrolvariableswhichweusetocontroltheplant.
Forexample,ifwedesignacontrollerforacarengine,theX(t)variablesarethedatawegetfromoursensors.
Theu(t)variableswouldincludethevalvesettingswhichweusetotrytocontroltheprocessofcombustion.
TheR(t)variablesprovideawayfortheneuralnetworkstorememberpasttime10Z-1X(t)R(t)X(t+1)u(t)TLRNFigure3:Timelaggedrecurrentnetwork(TLRN)XfyFigure4:Simultaneousrecurrentnetwork(SRN)11cycles,andtoimplicitlyestimateimportantvariableswhichcannotbeobserveddirectly.
Infact,theapplicationofTLRNstoautomobilecontrolisthemostvaluableapplicationofrecurrentnetworkseverdevelopedsofar.
Asimultaneousrecurrentnetwork(Figure4)isdenedasamapping:Y(t)=F(X(t),W)(3)whichiscomputedbyiteratingoverthefollowingequation:y(n+1)(t)=f(y(n)(t),X(t),W)(4)wherefissomesortoffeed-forwardnetworkorsystem,andYisdenedas:Y(t)=limn→∞y(n)(t)(5)WhenweuseYinthispaper,weusen=20insteadof∞here.
InFigure4,theoutputsoftheneuralnetworkcomebackagainasinputstothesamenetwork.
However,inconceptthereisnotimedelay.
Theinputsandoutputsshouldbesimultaneous.
Thatiswhyitiscalledasimultaneousrecurrentnetwork(SRN).
Inpractice,ofcourse,therewillalwaysbesomephys-icaltimedelaybetweentheoutputsandtheinputs.
HoweveriftheSRNisimplementedinfastcomputers,thistimedelaymaybeverysmallcomparedtothedelaybetweendierentframesofinputdata.
InFigure4,Xreferstotheinputdataatthecurrenttimeframet.
Thevectoryrepresentsthetemporaryoutputofthenetwork,whichisthenrecycledasanadditionalsetofinputstothenetwork.
AtthecenteroftheSRNisactuallythefeed-forwardnetworkwhichimplementsthefunctionf.
(IndesigninganSRN,youcanchooseanyfeed-forwardnetworkorsystemasyoulike.
Thefunctionfsimplydescribeswhichnetworkyouuse).
TheoutputoftheSRNatanytimetissimplythelimitofthetemporaryoutputy.
InEquation(3)and(4),noticethattherearetwointegers—nandt—whichcouldbothrepresentsomekindoftime.
Theintegertrepresentsaslowerkindoftimecycle,likethedelaybetweenframesofincomingdata.
Theintegernrepresentsafasterkindoftime,likethecomputingcycleofafastelectronicchip.
Forexample,ifwebuildacomputertoanalyzeimagescomingfromamoviecamera,"t"and"t+1"representtwosuccessiveincomingpictureswithamoviecamera.
Thereareusuallyonly32framespersecond.
(Inthehumanbrain,itseemsthatthereareonlyabout10framespersecondcomingintotheneocortex.
)Butifweuseafastneuralnetworkchip,thecomputationalcycle—thetimebetween"n"and"n+1"—couldbeassmallasamicrosecond.
Inactuality,itisnotnecessarytochoosebetweentime-laggedrecurrency(fromttot+1)andsimultaneousrecurrency(fromnton+1).
Itispossibletobuildahybridsystemwhichcontainsbothtypesofrecurrency.
Thiscouldbeveryusefulinanalyzingdatalikemoviepictures,whereweneedbothmemoryandsomeabilitytosegmenttheimages.
[9]discusseshowtobuildsuchahybrid.
12However,beforebuildingsuchahybrid,wemustrstlearntomakeSRNsworkbythemselves.
Finally,pleasenotethattheTLRNisnottheonlykindofneuralnetworkusedinpredictingdynamicalsystems.
Evenmorepopularisthetimedelayedneuralnetwork(TDNN),showninFigure5.
TheTDNNispopularbecauseitiseasytouse.
However,ithaslesscapability,inprinciple,becauseithasnoabilitytoestimateunknownvariables.
Itisespeciallyweakwhensomeofthesevariableschangeslowlyovertimeandrequirememorywhichpersistsoverlongtimeperiods.
Inaddition,theTLRNtstherequirementsofADPdirectly,whiletheTDNNdoesnot[9][16].
X(t)X(t-1)X(t-k)TDNNX(t+1)u(t-k)u(t-1)u(t)Figure5:Timedelayedneuralnetwork(TDNN)3.
3TrainingofSRNsandTLRNsTherearemanytypesoftrainingthathavebeenusedforrecurrentnetworks.
Dierenttypesoftraininggiverisetodierentkindsofcapabilitiesfordierenttasks.
ForthetaskswhichwehavedescribedfortheSRNandtheTLRN,theproperformsoftrainingallinvolvesomecalculationofthederivativesoferrorwithrespectstotheweights.
Usuallyafterthesederivativesareknown,theweightsareadaptedaccordingtoasimpleformulaasfollows:newWi,j=oldWi,jLRErrorWi,j(6)whereLRiscalledthelearningrate.
TherearevemainwaystotrainSRNs,allbasedondierentmethodsforcalculatingorapproximatingthederivatives.
FourofthesemethodscanalsobeusedwithTLRNs.
Somecanbeusedforcontrolapplications.
Butthedetailsofthoseapplicationsarebeyondthescopeofthispaper.
Theseve13typesoftrainingarelistedinFigure6.
Forthispaper,wehaveusedtwoofthesemethods:Backpropagationthroughtime(BTT)andTruncation.
TypesofSRNSimultaneousBackpropagationBackpropagationThroughTimeForwardPropagationTrainingTruncationErrorCriticsFigure6:TypesofSRNTrainingThevemethodsare:1.
Backpropagationthroughtime(BTT).
Thismethodandforwardpropa-gationarethetwomethodswhichcalculatethederivativesexactly.
BTTisalsolessexpensivethanforwardpropagation.
2.
Truncation.
Thisisthesimplestandleastexpensivemethod.
Itusesonlyonesimplepassofbackpropagationthroughthelastiterationofthemodel.
TruncationisprobablythemostpopularmethodusedtoadaptSRNseventhoughthepeoplewhouseitmostlyjustcallitordinarybackpropagation.
3.
Simultaneousbackpropagation.
Thisismorecomplexthantruncation,butitstillcanbeusedinrealtimelearning.
Itcalculatesderivativeswhichareexactintheneighborhoodofequilibriumbutitdoesnotaccountforthedetailsofthenetworkbeforeitreachestheneighborhoodofequilibrium.
4.
Errorcritics(showninFigure7).
ThisprovidesageneralapproximationtoBTTwhichissuitableforuseinreal-timelearning[9].
14TLRNTLRNCriticErrorCriticErrorErrorErrorR(t)X(t)u(t)X(t)u(t)X(t+1)X(t+1)X(t=1)u(t+1)R(t+1)λ^(t)λ^λ(t)(t+1)Figure7:ErrorCritics155.
Forwardpropagation.
This,likeBTT,calculatesexactderivatives.
Itisoftenconsideredsuitableforreal-timelearningbecausethecalculationsgoforwardintime.
However,whentherearenneuronsandmconnections,thecostofthismethodperunitoftimeisproportionaltonm.
Becauseofthishighcost,forwardpropagationisnotreallybrain-likeanymorethanBTT.
3.
3.
1Backpropagationthroughtime(BTT)BTTisageneralmethodforcalculatingallthederivativeofanyoutcomeorresultofaprocesswhichinvolvesrepeatedcallstothesamenetworkornet-worksusedtohelpcalculatesomekindofnaloutcomevariableorresultE.
Insomeapplications,Ecouldrepresentutility,performance,costorothersuchvariables.
Butinthispaper,Ewillbeusedtorepresenterror.
BTTwasrstproposedandimplementedin[17].
ThegeneralformofBTTisasfollows:fork=1toTdoforwardcalculation(k);calculateresultE;calculatedirectderivativesofEwithrespecttooutputsofforwardcalculations;fork=Tto1backpropagatethroughforwardscalculation(k),calculatingrun-ningtotalswhereappropriate.
ThesestepsareillustratedinFigure8.
Noticethatthisalgorithmcanbeappliedtoallkindsofcalculations.
ThuswecanapplyittocaseswherekrepresentsdataframestasintheTLRNs,ortocaseswherekrepresentsinternaliterationsnasintheSRNs.
AlsonotethateachboxofcalculationreceivesinputfromsomedashedlineswhichrepresentthederivativesofEwithrespecttotheoutputofthebox.
Inordertocalculatethederivativescomingoutofeachcalculationbox,onesimplyusesbackpropagationthroughthecalculationofthatboxstartingoutfromtheincomingderivatives.
WewillexplaininmoredetailhowthisworksintheSRNcaseandtheTLRNcase.
SofarasweknowBTThasbeenappliedinpublishedworkingsystemsforTLRNsandforcontrol,butnotyetforSRNsuntilnow.
However,Rumelhart,HintonandWilliams[18]didsuggestthatsomeoneshouldtrythis.
TheapplicationofBTTforTLRNsisdescribedatlengthin[2]and[9].
TheprocedureisillustratedinFigure9.
Inthisexamplethetotalerrorisactuallythesumoferrorovereachtimetwheretgoesfrom1toT.
ThereforetheoutputsoftheTLRNateachtimet(t#include#include#includevoidF_NET2(doubleF_Yhat,doubleW[30][30],doublex[30],intn,intm,intN,doubleF_W[30][30],doubleF_net[30],doubleF_Ws[30],doubleWs,doubleF_x[30]);voidNET(doubleW[30][30],doublex[30],doubleww[30],intn,intm,intN,doubleYhat[30]);41voidsynthesis(intB[30][30],intA[30][30],intn1,intn2);voidpweight(doubleWs,doubleF_Ws_T,doubleww[30],doubleF_net_T[30],doubleW[30][30],doubleF_W_T[30][30],intn,intN,intm);intminimum(ints,intt,intu,intv);intmin(intk,intl);doublef(doublex);voidmain(){inti,j,it,iz,ix,iy,lt,m,maxt,n,n1,n2,nn,nm,N,p,q,po,t,TT;intA[30][30],B[30][30];doublea,b,dot,e,e1,e2,es,mu,s,sum,F_Ws_T,Ws,F_Yhat,wi;doubleW[30][30],x[30],F_net_T[30],F_Ws[30],F_W_O[30][30],F_W[30][30];doubleF_W_T[30][30],F_net[30],ww[30],yy[21][12][8][8];doubleYhat[30],F_y[21][12][8][8],F_x[30],F_Jhat[30][30];doubleS_F_W1,S_F_W2,Lr_W,S_F_net1,S_F_net2,Lr_ww,Lr_Ws,F_Ws_O;doubley[50][50],F_net_O[50],F_Ws1,F_Ws2,W_O[50][50],ww_O[50],Ws_O;FILE*f;/*Numberofinputs,neuronsandoutput:7,3,1*//*'n'isthenumberoftheactiveneurons*//*'m'and'N'botharethenumberofinputs*//*'nm'isthenumberofmemoryis:5*//*'nn+1'*'nn+1'isthesizeofthemaze'*//*'TT'isthenumberoftrials*//*'lt'isthenumberoftheintervaltime*//*'maxt'isthemaxnumberforTinfigure[8]*//*Lr-Ws,Lr_wwandLr_WarethelearningratesforWs,wwandW*/a=0.
9;b=0.
2;n=5;m=11;N=11;nn=6;nm=5;TT=30000;lt=50;maxt=20;wi=25;Ws=40;e=0;po=pow(2,31)-1;/*InitialvaluesofOld*/F_Ws_O=1;for(i=m+1;i-1;q--){for(ix=0;ixS_F_W2)||(S_F_W1==S_F_W2))Lr_W=Lr_W*(a+b);elseif(S_F_W1S_F_net2)||(S_F_net1==S_F_net2))Lr_ww=Lr_ww*(a+b);elseif(S_F_net1F_Ws2)||(F_Ws1==F_Ws2))Lr_Ws=Lr_Ws*(a+b);elseif(F_Ws1m;i--){for(j=i+1;j0;i--)for(j=m+1;jl)r=l;elser=k;returnr;}voidpweight(doubleWs,doubleF_Ws_T,doubleww[30],doubleF_net_T[30],doubleW[30][30],doubleF_W_T[30][30],intn,intN,intm){inti,j;for(i=m+1;i
- arXiv:adap-org/9806001v1相关文档
- 优先级exhentai.org
- stratexhentai.org
- Componentexhentai.org
- elleexhentai.org
随风云-内蒙古三线BGP 2-2 5M 25/月 ，香港CN2 25/月 ，美国CERA 25/月 所有云服务器均支持5天无理由退款
公司成立于2021年，专注为用户提供低价高性能云计算产品，致力于云计算应用的易用性开发，面向全球客户提供基于云计算的IT解决方案与客户服务，拥有丰富的国内BGP、三线高防、香港等优质的IDC资源。公司一直秉承”以人为本、客户为尊、永续创新”的价值观，坚持”以微笑收获友善， 以尊重收获理解，以责任收获支持，以谦卑收获成长”的行为观向客户提供全面优质的互...
IMIDC(rainbow cloud)：香港/台湾/日本/莫斯科独立服务器特价,闪购大促销,最低30usd/月起
imidc怎么样?imidc彩虹网路，rainbow cloud知名服务器提供商。自营多地区数据中心，是 Apnic RIPE Afrinic Arin 认证服务商。拥有丰富的网路资源。 在2021年 6.18 开启了输血大促销，促销区域包括 香港 台湾 日本 莫斯科 等地促销机型为 E3係，参与促销地区有 香港 日本 台湾 莫斯科 等地， 限量50台,售罄为止,先到先得。所有服务器配置 CPU ...
Raksmart VPS主机如何设置取消自动续费
今天有看到Raksmart账户中有一台VPS主机即将到期，这台机器之前是用来测试评测使用的。这里有不打算续费，这不面对万一导致被自动续费忘记，所以我还是取消自动续费设置。如果我们也有类似的问题，这里就演示截图设置Raksmart取消自动续费。这里我们可以看到上图，在对应VPS主机的【其余操作】中可以看到默认已经是不自动续费，所以我们也不要担心被自动续费的。当然，如果有被自动续费，我们确实不想续费的...
exhentai.org为你推荐
-
特朗普取消访问丹麦特朗普出国访问什么飞机护送？刘祚天你们知道21世纪的DJ分为几种类型吗？（答对者重赏）www.522av.com跪求 我的三个母亲高清在线观看地址 我的三个母亲高清QVOD下载播放地址 我的三个母亲高清迅雷高速下载地址ip查询器查看自己IP的指令斗城网女追男有多易？喜欢你，可我不知道你喜不喜欢我！！平安夜希望有他陪我过ww.66bobo.com有的网址直接输入***.com就行了,不用WWW, 为什么?baqizi.cc孔融弑母是真的吗？woshiheida这个左下角水印woshiheida的gif出处在哪呢？急！！！！！梦遗姐昨晚和姐姐和她朋友一起吃晚饭，我们都喝了酒，我迷糊着回到家的，早上我回想起我好像发生关系射过，会不会是我姐姐，如果是这样我怎么办www.qqq147.comhttp://www.qq