desiredmimiai.net

mimiai.net  时间:2021-04-07  阅读:()
6.
034fNeuralNetNotesOctober28,2010Thesenotesareasupplementtomaterialpresentedinlecture.
Ilayoutthemathematicsmoreprettilyandextendtheanalysistohandlemultiple-neuronsperlayer.
Also,Idevelopthebackpropagationrule,whichisoftenneededonquizzes.
IuseanotationthatIthinkimprovesonpreviousexplanations.
Thereasonisthatthenotationhereplainlyassociateseachinput,output,andweightwithareadilyidentifiedneuron,aleft-sideoneandaright-sideone.
Whenyouarriveattheupdateformulas,youwillhavelesstroublerelatingthevariablesintheformulastothevariablesinadiagram.
Onetheotherhand,seeingyetanothernotationmayconfuseyou,soifyoualreadyfeelcomfortablewithasetofupdateformulas,youwillnotgainbyreadingthesenotes.
ThesigmoidfunctionThesigmoidfunction,y=1/(1+ex),isusedinsteadofastepfunctioninartificialneuralnetsbecausethesigmoidiscontinuous,whereasastepfunctionisnot,andyouneedcontinuitywheneveryouwanttousegradientascent.
Also,thesigmoidfunctionhasseveraldesirablequalities.
Forexample,thesigmoidfunction'svalue,y,approaches1asxbecomeshighlypositive;0asxbecomeshighlynegative;andequals1/2whenx=0.
Betteryet,thesigmoidfunctionfeaturesaremarkablysimplederivativeoftheoutput,y,withrespecttotheinput,x:dyd1=()dxdx1+exd=(1+ex)1dx=1*(1+ex)2*ex*11ex=*1+ex1+ex11+ex1=*1+ex1+ex11+ex11+ex1+ex1+ex=y(1y)Thus,remarkably,thederivativeoftheoutputwithrespecttotheinputisexpressedasasimplefunctionoftheoutput.
TheperformancefunctionThestandardperformancefunctionforgauginghowwellaneuralnetisdoingisgivenbythefollowing:1P=(dsampleosample)222wherePistheperformancefunction,dsampleisthedesiredoutputforsomespecificsampleandosampleistheobservedoutputforthatsample.
Fromthispointforward,assumethatdandoarethedesiredandobservedoutputsforaspecificsamplesothatweneednotdragasubscriptaroundasweworkthroughthealgebra.
ThereasonforchoosingthegivenformulaforPisthattheformulahasconvenientproperties.
Theformulayieldsamaximumato=dandmonotonicallydecreasesasodeviatesfromd.
Moreover,thederivativeofPwithrespecttooissimple:dPd1=[(do)2]dodo2=2*(do)1*12=doGradientascentBackpropagationisaspecializationoftheideaofgradientascent.
YouaretryingtofindthemaximumofaperformancefunctionP,bychangingtheweightsassociatedwithneurons,soyoumoveinthedirectionofthegradientinaspacethatgivesPasafunctionoftheweights,w.
Thatis,youmoveinthedirectionofmostrapidascentifwetakeastepinthedirectionwithcomponentsgovernedbythefollowingformula,whichshowshowmuchtochangeaweight,w,intermsofapartialderivative:PΔw∝wTheactualchangeisinuencedbyarateconstant,α;accordingly,thenewweight,w,isgivenbythefollowing:w=w+α*PwGradientdescentIftheperformancefunctionwere12(dsampleosample)2insteadof12(dsampleosample)2,thenyouwouldbesearchingfortheminimumratherthanthemaximumofP,andthechangeinwwouldbesubtractedfromwinsteadofadded,sowwouldbewα*wPinsteadofw+α*wP.
Thetwosignchanges,oneintheperformancefunctionandtheotherintheupdateformulacancel,sointheend,yougetthesameresultwhetheryouusegradientascent,asIprefer,orgradientdescent.
ThesimplestneuralnetConsiderthesimplestpossibleneuralnet:oneinput,oneoutput,andtwoneurons,theleftneuronandtherightneuron.
Anetwithtwoneuronsisthesmallestthatillustrateshowthederivativescanbecomputedlayerbylayer.
3xSigmoidWlplilolxSigmoidWrprorirLeftneuronRightneuronNotethatthesubscriptsindicatelayer.
Thus,il,wl,pl,andolaretheinput,weight,product,andoutputassociatedwiththeneuronontheleftwhileir,wr,pr,andoraretheinput,weight,product,andoutputassociatedwiththeneuronontheright.
Ofcourse,ol=ir.
Supposethattheoutputoftherightneuron,or,isthevaluethatdeterminesperformanceP.
TocomputethepartialderivativeofPwithrespecttotheweightintherightneuron,wr,youneedthechainrule,whichallowsyoutocomputepartialderivativesofonevariablewithrespecttoanotherintermsofanintermediatevariable.
Inparticular,forwr,youhavethefollowing,takingortobetheintermediatevariable:PPor=*wrorwrNow,youcanrepeat,usingthechain-ruletoturnworrintooprr*wprr:PPorpr=**wrorprwrConveniently,youhaveseentwoofthederivativesalready,andthethird,wprr=(wrw*rol),iseasytocompute:P=[(dor)]*[or(1or)]*[ir]wrRepeatingtheanalysisforwlyieldsthefollowing.
Eachlineisthesameasthepreviously,exceptthatonemorepartialderivativeisexpandedusingthechainrule:P=P*orwlorwl=P*or*prorprwl=P*or*pr*olorprolwl=P*or*pr*ol*plorprolplwl=[(dor)]*[or(1or)]*[wr]*[ol(1ol)]*[il]4Thus,thederivativeconsistsofproductsoftermsthathavealreadybeencomputedandtermsinthevicinityofwl.
Thisisclearerifyouwritethetwoderivativesnexttooneanother:P=(dor)*or(1or)*irwrP=(dor)*or(1or)*wr*ol(1ol)*ilwlYoucansimplifytheequationsbydefiningδsasfollows,whereeachdeltaisassociatedwitheithertheleftorrightneuron:δr=or(1or)*(dor)δl=ol(1ol)*wr*δrThen,youcanwritethepartialderivativeswiththeδs:P=ir*δrwrP=il*δlwlIfyouaddmorelayerstothefrontofthenetwork,eachweighthasapartialderivativesthatiscomputedlikethepartialderivativeoftheweightoftheleftneuron.
Thatis,eachhasapartialderivativedeterminedbyitsinputanditsdelta,whereitsdeltainturnisdeterminedbyitsoutput,theweighttoitsright,andthedeltatoitsright.
Thus,fortheweightsinthefinallayer,youcomputethechangeasfollows,whereIusefasthesubscriptinsteadofrtoemphasizethatthecomputationisfortheneuroninthefinallayer:Δwf=α*if*δfwhereδf=of(1of)*(dof)Forallotherlayers,youcomputethechangeasfollows:Δwl=α*il*δlwhereδl=ol(1ol)*wr*δrMoreneuronsperlayersOfcourse,youreallywantbackpropagationformulasfornotonlyanynumberoflayersbutalsoforanynumberofneuronsperlayer,eachofwhichcanhavemultipleinputs,eachwithitsownweight.
Accordingly,youneedtogeneralizeinanotherdirection,allowingmultipleneuronsineachlayerandmultipleweightsattachedtoeachneuron.
Thegeneralizationisanadventureinsummations,withlotsofsubscriptstokeepstraight,butintheend,theresultmatchesintuition.
Forthefinallayer,theremaybemanyneurons,sotheformula'sneedanindex,k,indicatingwhichfinalnodeneuronisinplay.
Foranyweightcontained5inthefinal-layerneuron,fk,youcomputethechangeasfollowsfromtheinputcorrespondingtotheweightandfromtheδassociatedwiththeneuron:Δw=α*i*δfkδfk=ofk(1ofk)*(dkofk)Notethattheoutputofeachfinal-layerneuronoutputissubtractedfromtheoutputdesiredforthatneuron.
Forotherlayers,theremayalsobemanyneurons,andtheoutputofeachmayinuencealltheneuronsinthenextlayertotheright.
Thechangeinweighthastoaccountforwhathappenstoallofthoseneuronstotheright,soasummationappears,butotherwiseyoucomputethechange,asbefore,fromtheinputcorrespondingtotheweightandfromtheδassociatedwiththeneuron:Δw=α*i*δliδli=oli(1oli)*wli→rj*δrjjNotethatwli→rjistheweightthatconnectsthejthright-sideneurontotheoutputoftheithleft-sideneuron.
SummaryOnceyouunderstoodhowtoderivetheformulas,youcancombineandsimplifytheminpreparationforsolvingproblems.
Foreachweight,youcomputetheweight'schangefromtheinputcorrespondingtotheweightandfromtheδassociatedwiththeneuron.
Assumingthatδisthedeltaassociatedwiththatneuron,youhavethefollowing,wherew→rjistheweightconnectingtheoutputoftheneuronyouareworkingon,theithleft-sideneuron,tothejthright-sideneuron,andδrjistheδassociatedwiththatright-sideneuron.
δo=o(1o)*(do)forthefinallayerδli=oli(1oli)*wli→rj*δrjotherwisejThatis,youcomputedchangeinaneuron'sw,ineverylayer,bymultiplyingαtimestheneuron'sinputtimesitsδ.
Theδisdeterminedforallbutthefinallayerintermsoftheneuron'soutputandalltheweightsthatconnectthatoutputtoneuronsinthelayertotherightandtheδsassociatedwiththoseright-sideneurons.
Theδforeachneuroninthefinallayerisdeterminedonlybytheoutputofthatneuronandbythedifferencebetweenthedesiredoutputandtheactualoutputofthatneuron.
6WeightsanddeltasinlayertotherightNeuronwithweighttobeadjustedw→r1wxoixxΣ∫w→rNWeighttobeadjustedxxxΣ∫δ1xxxΣ∫δΝMITOpenCourseWarehttp://ocw.
mit.
edu6.
034ArtificialIntelligenceFall2010ForinformationaboutcitingthesematerialsorourTermsofUse,visit:http://ocw.
mit.
edu/terms.

LOCVPS洛杉矶CN2线路KVM上线,洛杉矶/香港云地/香港邦联7折

LOCVPS发来了新的洛杉矶CN2线路主机上线通知,基于KVM架构,目前可与香港云地、香港邦联机房XEN架构主机一起适用7折优惠码,优惠后最低美国洛杉矶CN2线路KVM架构2GB内存套餐月付38.5元起。LOCPVS是一家成立较早的国人VPS服务商,目前提供洛杉矶MC、洛杉矶C3、和香港邦联、香港沙田电信、香港大埔、日本东京、日本大阪、新加坡、德国和荷兰等机房VPS主机,基于KVM或者XEN架构。...

Megalayer 香港CN2优化线路VPS主机速度和性能综合评测

对于Megalayer云服务器提供商在之前也有对于他们家的美国服务器和香港服务器进行过评测和介绍,但是对于大部分网友来说需要独立服务器和站群服务器并不是特别的普及,我们很多网友使用较多的还是云服务器或者VPS主机比较多。在前面也有在"Megalayer新增香港VPS主机 1GB内存 50GB SSD 2M带宽 月59元"文章中有介绍到Megalayer商家有新增香港CN2优化VPS主机。那时候看这...

10gbiz七月活动首月半价$2.36/月: 香港/洛杉矶CN2 GIA VPS

10gbiz怎么样?10gbiz 美国万兆带宽供应商,主打美国直连大带宽,真实硬防。除美国外还提供线路非常优质的香港、日本等数据中心可供选择,全部机房均支持增加独立硬防。洛杉矶特色线路去程三网直连(电信、联通、移动)回程CN2 GIA优化,全天低延迟。中国大陆访问质量优秀,最多可增加至600G硬防。香港七星级网络,去程回程均为电信CN2 GIA+联通+移动,大陆访问相较其他香港GIA线路平均速度更...

mimiai.net为你推荐
美国互联网瘫痪美国掐断中国互联网怎么办,我们如何解决?是否有后招?百度关键词价格查询百度推广关键词怎么扣费?www.javmoo.comjavimdb怎么看66smsm.comffff66com手机可以观看视频吗?机器蜘蛛《不思议迷宫》四个机器蜘蛛怎么得 获得攻略方法介绍xvideos..comxvideos 怎么下载bk乐乐《哭泣的Bk》是Bk乐乐唱的吗?dpscycle痛苦术士PVE输出宏222cc.com怎样开通网站啊月风随笔散文校园月色600字初中作文
美国vps租用 大庆服务器租用 免费注册网站域名 2019年感恩节 bash漏洞 火车票抢票攻略 hostker 免费申请个人网站 百度云1t 支持外链的相册 彩虹云 论坛主机 广东主机托管 web是什么意思 免费网站加速 qq空间打开很慢 server2008 卡巴斯基免费版下载 卡巴斯基免费版 vim 更多