desiredmimiai.net

mimiai.net  时间:2021-04-07  阅读:()
6.
034fNeuralNetNotesOctober28,2010Thesenotesareasupplementtomaterialpresentedinlecture.
Ilayoutthemathematicsmoreprettilyandextendtheanalysistohandlemultiple-neuronsperlayer.
Also,Idevelopthebackpropagationrule,whichisoftenneededonquizzes.
IuseanotationthatIthinkimprovesonpreviousexplanations.
Thereasonisthatthenotationhereplainlyassociateseachinput,output,andweightwithareadilyidentifiedneuron,aleft-sideoneandaright-sideone.
Whenyouarriveattheupdateformulas,youwillhavelesstroublerelatingthevariablesintheformulastothevariablesinadiagram.
Onetheotherhand,seeingyetanothernotationmayconfuseyou,soifyoualreadyfeelcomfortablewithasetofupdateformulas,youwillnotgainbyreadingthesenotes.
ThesigmoidfunctionThesigmoidfunction,y=1/(1+ex),isusedinsteadofastepfunctioninartificialneuralnetsbecausethesigmoidiscontinuous,whereasastepfunctionisnot,andyouneedcontinuitywheneveryouwanttousegradientascent.
Also,thesigmoidfunctionhasseveraldesirablequalities.
Forexample,thesigmoidfunction'svalue,y,approaches1asxbecomeshighlypositive;0asxbecomeshighlynegative;andequals1/2whenx=0.
Betteryet,thesigmoidfunctionfeaturesaremarkablysimplederivativeoftheoutput,y,withrespecttotheinput,x:dyd1=()dxdx1+exd=(1+ex)1dx=1*(1+ex)2*ex*11ex=*1+ex1+ex11+ex1=*1+ex1+ex11+ex11+ex1+ex1+ex=y(1y)Thus,remarkably,thederivativeoftheoutputwithrespecttotheinputisexpressedasasimplefunctionoftheoutput.
TheperformancefunctionThestandardperformancefunctionforgauginghowwellaneuralnetisdoingisgivenbythefollowing:1P=(dsampleosample)222wherePistheperformancefunction,dsampleisthedesiredoutputforsomespecificsampleandosampleistheobservedoutputforthatsample.
Fromthispointforward,assumethatdandoarethedesiredandobservedoutputsforaspecificsamplesothatweneednotdragasubscriptaroundasweworkthroughthealgebra.
ThereasonforchoosingthegivenformulaforPisthattheformulahasconvenientproperties.
Theformulayieldsamaximumato=dandmonotonicallydecreasesasodeviatesfromd.
Moreover,thederivativeofPwithrespecttooissimple:dPd1=[(do)2]dodo2=2*(do)1*12=doGradientascentBackpropagationisaspecializationoftheideaofgradientascent.
YouaretryingtofindthemaximumofaperformancefunctionP,bychangingtheweightsassociatedwithneurons,soyoumoveinthedirectionofthegradientinaspacethatgivesPasafunctionoftheweights,w.
Thatis,youmoveinthedirectionofmostrapidascentifwetakeastepinthedirectionwithcomponentsgovernedbythefollowingformula,whichshowshowmuchtochangeaweight,w,intermsofapartialderivative:PΔw∝wTheactualchangeisinuencedbyarateconstant,α;accordingly,thenewweight,w,isgivenbythefollowing:w=w+α*PwGradientdescentIftheperformancefunctionwere12(dsampleosample)2insteadof12(dsampleosample)2,thenyouwouldbesearchingfortheminimumratherthanthemaximumofP,andthechangeinwwouldbesubtractedfromwinsteadofadded,sowwouldbewα*wPinsteadofw+α*wP.
Thetwosignchanges,oneintheperformancefunctionandtheotherintheupdateformulacancel,sointheend,yougetthesameresultwhetheryouusegradientascent,asIprefer,orgradientdescent.
ThesimplestneuralnetConsiderthesimplestpossibleneuralnet:oneinput,oneoutput,andtwoneurons,theleftneuronandtherightneuron.
Anetwithtwoneuronsisthesmallestthatillustrateshowthederivativescanbecomputedlayerbylayer.
3xSigmoidWlplilolxSigmoidWrprorirLeftneuronRightneuronNotethatthesubscriptsindicatelayer.
Thus,il,wl,pl,andolaretheinput,weight,product,andoutputassociatedwiththeneuronontheleftwhileir,wr,pr,andoraretheinput,weight,product,andoutputassociatedwiththeneuronontheright.
Ofcourse,ol=ir.
Supposethattheoutputoftherightneuron,or,isthevaluethatdeterminesperformanceP.
TocomputethepartialderivativeofPwithrespecttotheweightintherightneuron,wr,youneedthechainrule,whichallowsyoutocomputepartialderivativesofonevariablewithrespecttoanotherintermsofanintermediatevariable.
Inparticular,forwr,youhavethefollowing,takingortobetheintermediatevariable:PPor=*wrorwrNow,youcanrepeat,usingthechain-ruletoturnworrintooprr*wprr:PPorpr=**wrorprwrConveniently,youhaveseentwoofthederivativesalready,andthethird,wprr=(wrw*rol),iseasytocompute:P=[(dor)]*[or(1or)]*[ir]wrRepeatingtheanalysisforwlyieldsthefollowing.
Eachlineisthesameasthepreviously,exceptthatonemorepartialderivativeisexpandedusingthechainrule:P=P*orwlorwl=P*or*prorprwl=P*or*pr*olorprolwl=P*or*pr*ol*plorprolplwl=[(dor)]*[or(1or)]*[wr]*[ol(1ol)]*[il]4Thus,thederivativeconsistsofproductsoftermsthathavealreadybeencomputedandtermsinthevicinityofwl.
Thisisclearerifyouwritethetwoderivativesnexttooneanother:P=(dor)*or(1or)*irwrP=(dor)*or(1or)*wr*ol(1ol)*ilwlYoucansimplifytheequationsbydefiningδsasfollows,whereeachdeltaisassociatedwitheithertheleftorrightneuron:δr=or(1or)*(dor)δl=ol(1ol)*wr*δrThen,youcanwritethepartialderivativeswiththeδs:P=ir*δrwrP=il*δlwlIfyouaddmorelayerstothefrontofthenetwork,eachweighthasapartialderivativesthatiscomputedlikethepartialderivativeoftheweightoftheleftneuron.
Thatis,eachhasapartialderivativedeterminedbyitsinputanditsdelta,whereitsdeltainturnisdeterminedbyitsoutput,theweighttoitsright,andthedeltatoitsright.
Thus,fortheweightsinthefinallayer,youcomputethechangeasfollows,whereIusefasthesubscriptinsteadofrtoemphasizethatthecomputationisfortheneuroninthefinallayer:Δwf=α*if*δfwhereδf=of(1of)*(dof)Forallotherlayers,youcomputethechangeasfollows:Δwl=α*il*δlwhereδl=ol(1ol)*wr*δrMoreneuronsperlayersOfcourse,youreallywantbackpropagationformulasfornotonlyanynumberoflayersbutalsoforanynumberofneuronsperlayer,eachofwhichcanhavemultipleinputs,eachwithitsownweight.
Accordingly,youneedtogeneralizeinanotherdirection,allowingmultipleneuronsineachlayerandmultipleweightsattachedtoeachneuron.
Thegeneralizationisanadventureinsummations,withlotsofsubscriptstokeepstraight,butintheend,theresultmatchesintuition.
Forthefinallayer,theremaybemanyneurons,sotheformula'sneedanindex,k,indicatingwhichfinalnodeneuronisinplay.
Foranyweightcontained5inthefinal-layerneuron,fk,youcomputethechangeasfollowsfromtheinputcorrespondingtotheweightandfromtheδassociatedwiththeneuron:Δw=α*i*δfkδfk=ofk(1ofk)*(dkofk)Notethattheoutputofeachfinal-layerneuronoutputissubtractedfromtheoutputdesiredforthatneuron.
Forotherlayers,theremayalsobemanyneurons,andtheoutputofeachmayinuencealltheneuronsinthenextlayertotheright.
Thechangeinweighthastoaccountforwhathappenstoallofthoseneuronstotheright,soasummationappears,butotherwiseyoucomputethechange,asbefore,fromtheinputcorrespondingtotheweightandfromtheδassociatedwiththeneuron:Δw=α*i*δliδli=oli(1oli)*wli→rj*δrjjNotethatwli→rjistheweightthatconnectsthejthright-sideneurontotheoutputoftheithleft-sideneuron.
SummaryOnceyouunderstoodhowtoderivetheformulas,youcancombineandsimplifytheminpreparationforsolvingproblems.
Foreachweight,youcomputetheweight'schangefromtheinputcorrespondingtotheweightandfromtheδassociatedwiththeneuron.
Assumingthatδisthedeltaassociatedwiththatneuron,youhavethefollowing,wherew→rjistheweightconnectingtheoutputoftheneuronyouareworkingon,theithleft-sideneuron,tothejthright-sideneuron,andδrjistheδassociatedwiththatright-sideneuron.
δo=o(1o)*(do)forthefinallayerδli=oli(1oli)*wli→rj*δrjotherwisejThatis,youcomputedchangeinaneuron'sw,ineverylayer,bymultiplyingαtimestheneuron'sinputtimesitsδ.
Theδisdeterminedforallbutthefinallayerintermsoftheneuron'soutputandalltheweightsthatconnectthatoutputtoneuronsinthelayertotherightandtheδsassociatedwiththoseright-sideneurons.
Theδforeachneuroninthefinallayerisdeterminedonlybytheoutputofthatneuronandbythedifferencebetweenthedesiredoutputandtheactualoutputofthatneuron.
6WeightsanddeltasinlayertotherightNeuronwithweighttobeadjustedw→r1wxoixxΣ∫w→rNWeighttobeadjustedxxxΣ∫δ1xxxΣ∫δΝMITOpenCourseWarehttp://ocw.
mit.
edu6.
034ArtificialIntelligenceFall2010ForinformationaboutcitingthesematerialsorourTermsofUse,visit:http://ocw.
mit.
edu/terms.

欧路云:美国200G高防云-10元/月,香港云-15元/月,加拿大480G高防云-23元/月

欧路云 主要运行弹性云服务器,可自由定制配置,可选加拿大的480G超高防系列,也可以选择美国(200G高防)系列,也有速度直逼内地的香港CN2系列。所有配置都可以在下单的时候自行根据项目 需求来定制自由升级降级 (降级按天数配置费用 退款回预存款)。由专业人员提供一系列的技术支持!官方网站:https://www.oulucloud.com/云服务器(主机测评专属优惠)全场8折 优惠码:zhuji...

hostio荷兰10Gbps带宽,10Gbps带宽,€5/月,最低配2G内存+2核+5T流量

成立于2006年的荷兰Access2.IT Group B.V.(可查:VAT: NL853006404B01,CoC: 58365400) 一直运作着主机周边的业务,当前正在对荷兰的高性能AMD平台的VPS进行5折优惠,所有VPS直接砍一半。自有AS208258,vps母鸡配置为Supermicro 1024US-TRT 1U,2*AMD Epyc 7452(64核128线程),16条32G D...

新网,域名7月盛夏1核心2G内存.COM域名仅19.9元/首年,主机9.9元/月,企业邮箱0元体验

新网好不好?新网域名便宜吗?新网怎么样?新网是国内老牌知名域名注册商,企业正规化运营,资质齐全,与阿里云万网和腾讯云DNSPOD同为国内服务商巨头。近日新网发布了最新的七月放价季优惠活动,主要针对域名、云主机、企业邮箱、SSL证书等多款云产品推送了超值的优惠,其中.com顶级域名仅19.9元/首年,.cn域名仅16元/首年,云主机1核心2G内存3Mbps带宽仅9.9元/月,企业邮箱更是免费送1年,...

mimiai.net为你推荐
乐划锁屏乐视手机怎么换锁屏壁纸??硬盘的工作原理硬盘的工作原理?是怎样存取数据的?蓝色骨头手机宠物的一个蓝色骨头代表多少级,灰色又代表多少级,另外假如有骨头又代表多少级嘉兴商标注册我想注册个商标怎么注册啊?百度关键词价格查询百度关键词排名价格是多少haokandianyingwang谁有好看电影网站啊、要无毒播放速度快的、在线等www.5any.comwww.qbo5.com 这个网站要安装播放器杨丽晓博客杨丽晓哪一年出生的?dadi.tv电视机如何从iptv转换成tv?555sss.com拜求:http://www.jjj555.com/这个网站是用的什么程序
yaokan永久域名经常更换 西部数码vps 域名停靠一青草视频 高防dns 电影服务器 cloudstack 20g硬盘 免费网站监控 360抢票助手 云主机51web 嘟牛 工作站服务器 网站卫士 能外链的相册 最漂亮的qq空间 smtp虚拟服务器 网购分享 ebay注册 什么是web服务器 免费asp空间申请 更多