6.
034fNeuralNetNotesOctober28,2010Thesenotesareasupplementtomaterialpresentedinlecture.
Ilayoutthemathematicsmoreprettilyandextendtheanalysistohandlemultiple-neuronsperlayer.
Also,Idevelopthebackpropagationrule,whichisoftenneededonquizzes.
IuseanotationthatIthinkimprovesonpreviousexplanations.
Thereasonisthatthenotationhereplainlyassociateseachinput,output,andweightwithareadilyidentifiedneuron,aleft-sideoneandaright-sideone.
Whenyouarriveattheupdateformulas,youwillhavelesstroublerelatingthevariablesintheformulastothevariablesinadiagram.
Onetheotherhand,seeingyetanothernotationmayconfuseyou,soifyoualreadyfeelcomfortablewithasetofupdateformulas,youwillnotgainbyreadingthesenotes.
ThesigmoidfunctionThesigmoidfunction,y=1/(1+ex),isusedinsteadofastepfunctioninartificialneuralnetsbecausethesigmoidiscontinuous,whereasastepfunctionisnot,andyouneedcontinuitywheneveryouwanttousegradientascent.
Also,thesigmoidfunctionhasseveraldesirablequalities.
Forexample,thesigmoidfunction'svalue,y,approaches1asxbecomeshighlypositive;0asxbecomeshighlynegative;andequals1/2whenx=0.
Betteryet,thesigmoidfunctionfeaturesaremarkablysimplederivativeoftheoutput,y,withrespecttotheinput,x:dyd1=()dxdx1+exd=(1+ex)1dx=1*(1+ex)2*ex*11ex=*1+ex1+ex11+ex1=*1+ex1+ex11+ex11+ex1+ex1+ex=y(1y)Thus,remarkably,thederivativeoftheoutputwithrespecttotheinputisexpressedasasimplefunctionoftheoutput.
TheperformancefunctionThestandardperformancefunctionforgauginghowwellaneuralnetisdoingisgivenbythefollowing:1P=(dsampleosample)222wherePistheperformancefunction,dsampleisthedesiredoutputforsomespecificsampleandosampleistheobservedoutputforthatsample.
Fromthispointforward,assumethatdandoarethedesiredandobservedoutputsforaspecificsamplesothatweneednotdragasubscriptaroundasweworkthroughthealgebra.
ThereasonforchoosingthegivenformulaforPisthattheformulahasconvenientproperties.
Theformulayieldsamaximumato=dandmonotonicallydecreasesasodeviatesfromd.
Moreover,thederivativeofPwithrespecttooissimple:dPd1=[(do)2]dodo2=2*(do)1*12=doGradientascentBackpropagationisaspecializationoftheideaofgradientascent.
YouaretryingtofindthemaximumofaperformancefunctionP,bychangingtheweightsassociatedwithneurons,soyoumoveinthedirectionofthegradientinaspacethatgivesPasafunctionoftheweights,w.
Thatis,youmoveinthedirectionofmostrapidascentifwetakeastepinthedirectionwithcomponentsgovernedbythefollowingformula,whichshowshowmuchtochangeaweight,w,intermsofapartialderivative:PΔw∝wTheactualchangeisinuencedbyarateconstant,α;accordingly,thenewweight,w,isgivenbythefollowing:w=w+α*PwGradientdescentIftheperformancefunctionwere12(dsampleosample)2insteadof12(dsampleosample)2,thenyouwouldbesearchingfortheminimumratherthanthemaximumofP,andthechangeinwwouldbesubtractedfromwinsteadofadded,sowwouldbewα*wPinsteadofw+α*wP.
Thetwosignchanges,oneintheperformancefunctionandtheotherintheupdateformulacancel,sointheend,yougetthesameresultwhetheryouusegradientascent,asIprefer,orgradientdescent.
ThesimplestneuralnetConsiderthesimplestpossibleneuralnet:oneinput,oneoutput,andtwoneurons,theleftneuronandtherightneuron.
Anetwithtwoneuronsisthesmallestthatillustrateshowthederivativescanbecomputedlayerbylayer.
3xSigmoidWlplilolxSigmoidWrprorirLeftneuronRightneuronNotethatthesubscriptsindicatelayer.
Thus,il,wl,pl,andolaretheinput,weight,product,andoutputassociatedwiththeneuronontheleftwhileir,wr,pr,andoraretheinput,weight,product,andoutputassociatedwiththeneuronontheright.
Ofcourse,ol=ir.
Supposethattheoutputoftherightneuron,or,isthevaluethatdeterminesperformanceP.
TocomputethepartialderivativeofPwithrespecttotheweightintherightneuron,wr,youneedthechainrule,whichallowsyoutocomputepartialderivativesofonevariablewithrespecttoanotherintermsofanintermediatevariable.
Inparticular,forwr,youhavethefollowing,takingortobetheintermediatevariable:PPor=*wrorwrNow,youcanrepeat,usingthechain-ruletoturnworrintooprr*wprr:PPorpr=**wrorprwrConveniently,youhaveseentwoofthederivativesalready,andthethird,wprr=(wrw*rol),iseasytocompute:P=[(dor)]*[or(1or)]*[ir]wrRepeatingtheanalysisforwlyieldsthefollowing.
Eachlineisthesameasthepreviously,exceptthatonemorepartialderivativeisexpandedusingthechainrule:P=P*orwlorwl=P*or*prorprwl=P*or*pr*olorprolwl=P*or*pr*ol*plorprolplwl=[(dor)]*[or(1or)]*[wr]*[ol(1ol)]*[il]4Thus,thederivativeconsistsofproductsoftermsthathavealreadybeencomputedandtermsinthevicinityofwl.
Thisisclearerifyouwritethetwoderivativesnexttooneanother:P=(dor)*or(1or)*irwrP=(dor)*or(1or)*wr*ol(1ol)*ilwlYoucansimplifytheequationsbydefiningδsasfollows,whereeachdeltaisassociatedwitheithertheleftorrightneuron:δr=or(1or)*(dor)δl=ol(1ol)*wr*δrThen,youcanwritethepartialderivativeswiththeδs:P=ir*δrwrP=il*δlwlIfyouaddmorelayerstothefrontofthenetwork,eachweighthasapartialderivativesthatiscomputedlikethepartialderivativeoftheweightoftheleftneuron.
Thatis,eachhasapartialderivativedeterminedbyitsinputanditsdelta,whereitsdeltainturnisdeterminedbyitsoutput,theweighttoitsright,andthedeltatoitsright.
Thus,fortheweightsinthefinallayer,youcomputethechangeasfollows,whereIusefasthesubscriptinsteadofrtoemphasizethatthecomputationisfortheneuroninthefinallayer:Δwf=α*if*δfwhereδf=of(1of)*(dof)Forallotherlayers,youcomputethechangeasfollows:Δwl=α*il*δlwhereδl=ol(1ol)*wr*δrMoreneuronsperlayersOfcourse,youreallywantbackpropagationformulasfornotonlyanynumberoflayersbutalsoforanynumberofneuronsperlayer,eachofwhichcanhavemultipleinputs,eachwithitsownweight.
Accordingly,youneedtogeneralizeinanotherdirection,allowingmultipleneuronsineachlayerandmultipleweightsattachedtoeachneuron.
Thegeneralizationisanadventureinsummations,withlotsofsubscriptstokeepstraight,butintheend,theresultmatchesintuition.
Forthefinallayer,theremaybemanyneurons,sotheformula'sneedanindex,k,indicatingwhichfinalnodeneuronisinplay.
Foranyweightcontained5inthefinal-layerneuron,fk,youcomputethechangeasfollowsfromtheinputcorrespondingtotheweightandfromtheδassociatedwiththeneuron:Δw=α*i*δfkδfk=ofk(1ofk)*(dkofk)Notethattheoutputofeachfinal-layerneuronoutputissubtractedfromtheoutputdesiredforthatneuron.
Forotherlayers,theremayalsobemanyneurons,andtheoutputofeachmayinuencealltheneuronsinthenextlayertotheright.
Thechangeinweighthastoaccountforwhathappenstoallofthoseneuronstotheright,soasummationappears,butotherwiseyoucomputethechange,asbefore,fromtheinputcorrespondingtotheweightandfromtheδassociatedwiththeneuron:Δw=α*i*δliδli=oli(1oli)*wli→rj*δrjjNotethatwli→rjistheweightthatconnectsthejthright-sideneurontotheoutputoftheithleft-sideneuron.
SummaryOnceyouunderstoodhowtoderivetheformulas,youcancombineandsimplifytheminpreparationforsolvingproblems.
Foreachweight,youcomputetheweight'schangefromtheinputcorrespondingtotheweightandfromtheδassociatedwiththeneuron.
Assumingthatδisthedeltaassociatedwiththatneuron,youhavethefollowing,wherew→rjistheweightconnectingtheoutputoftheneuronyouareworkingon,theithleft-sideneuron,tothejthright-sideneuron,andδrjistheδassociatedwiththatright-sideneuron.
δo=o(1o)*(do)forthefinallayerδli=oli(1oli)*wli→rj*δrjotherwisejThatis,youcomputedchangeinaneuron'sw,ineverylayer,bymultiplyingαtimestheneuron'sinputtimesitsδ.
Theδisdeterminedforallbutthefinallayerintermsoftheneuron'soutputandalltheweightsthatconnectthatoutputtoneuronsinthelayertotherightandtheδsassociatedwiththoseright-sideneurons.
Theδforeachneuroninthefinallayerisdeterminedonlybytheoutputofthatneuronandbythedifferencebetweenthedesiredoutputandtheactualoutputofthatneuron.
6WeightsanddeltasinlayertotherightNeuronwithweighttobeadjustedw→r1wxoixxΣ∫w→rNWeighttobeadjustedxxxΣ∫δ1xxxΣ∫δΝMITOpenCourseWarehttp://ocw.
mit.
edu6.
034ArtificialIntelligenceFall2010ForinformationaboutcitingthesematerialsorourTermsofUse,visit:http://ocw.
mit.
edu/terms.
JUSTG,这个主机商第二个接触到,之前是有介绍到有提供俄罗斯CN2 GIA VPS主机活动的,商家成立时间不久看信息是2020年,公司隶属于一家叫AFRICA CLOUD LIMITED的公司,提供的产品为基于KVM架构VPS主机,数据中心在非洲(南非)、俄罗斯(莫斯科),国内访问双向CN2,线路质量不错。有很多服务商实际上都是国人背景的,有的用英文、繁体搭建的冒充老外,这个服务商不清楚是不是真...
wordpress高级跨屏企业主题,通用响应式跨平台站点开发,自适应PC端+各移动端屏幕设备,高级可视化自定义设置模块+高效的企业站搜索优化。wordpress绿色企业自适应主题采用标准的HTML5+CSS3语言开发,兼容当下的各种主流浏览器: IE 6+(以及类似360、遨游等基于IE内核的)、Firefox、Google Chrome、Safari、Opera等;同时支持移动终端的常用浏览器应...
提速啦的来历提速啦是 网站 本着“良心 便宜 稳定”的初衷 为小白用户避免被坑 由赣州王成璟网络科技有限公司旗下赣州提速啦网络科技有限公司运营 投资1000万人民币 在美国Cera 香港CTG 香港Cera 国内 杭州 宿迁 浙江 赣州 南昌 大连 辽宁 扬州 等地区建立数据中心 正规持有IDC ISP CDN 云牌照 公司。公司购买产品支持3天内退款 超过3天步退款政策。提速啦的市场定位提速啦主...
mimiai.net为你推荐
小度商城小度智能音箱1s上面的黄圈不熄灭怎么回事,第一天还能熄灭permissiondeniedpermission denied 怎么解决johncusack约翰·库萨克好看的的恐怖片全集关键字关键词编故事lunwenjiance论文检测,知网的是32.4%,改了以后,维普的是29.23%。如果再到知网查,会不会超过呢?haole018.com为什么www.haole008.com在我这里打不开啊,是不是haole008换新的地址了?8090lu.com8090看看电影网怎么打不开了javmoo.com找下载JAV软件格式的网站qq530.com求教:如何下载http://www.qq530.com/ 上的音乐partnersonline国内有哪些知名的ACCA培训机构
域名网站 怎么注册域名 高防服务器租用qy 泛域名解析 国外永久服务器 virpus webhosting 512av 好看的留言 万网空间购买 国外视频网站有哪些 太原联通测速 php服务器 privatetracker 八度空间论坛 护卫神主机管理系统 网站服务器硬件配置 文件服务器硬件配置 电脑主机结构图 1000元电脑主机配置 更多