神经算法BP算法的BP算法基本介绍

神经算法  时间:2021-07-02  阅读:()

BP算法及其改进

传统的BP算法及其改进算法的一个很大缺点是:由于其误差目标函数对于待学习的连接权值来说非凸的,存在局部最小点,对网络进行训练时,这些算法的权值一旦落入权值空间的局部最小点就很难跳出,因而无法达到全局最小点(即最优点)而使得网络训练失败。

针对这些缺陷,根据凸函数及其共轭的性质,利用Fenchel不等式,使用约束优化理论中的罚函数方法构造出了带有惩罚项的新误差目标函数。

用新的目标函数对前馈神经网络进行优化训练时,隐层输出也作为被优化变量。

这个目标函数的主要特点有: 1.固定隐层输出,该目标函数对连接权值来说是凸的;固定连接权值,对隐层输出来说是凸的。

这样在对连接权值和隐层输出进行交替优化时,它们所面对的目标函数都是凸函数,不存在局部最小的问题,算法对于初始权值的敏感性降低; 2.由于惩罚因子是逐渐增大的,使得权值的搜索空间变得比较大,从而对于大规模的网络也能够训练,在一定程度上降低了训练过程陷入局部最小的可能性。

这些特性能够在很大程度上有效地克服以往前馈网络的训练算法易于陷入局部最小而使网络训练失败的重大缺陷,也为利用凸优化理论研究前馈神经网络的学习算法开创了一个新思路。

在网络训练时,可以对连接权值和隐层输出进行交替优化。

把这种新算法应用到前馈神经网络训练学习中,在学习速度、泛化能力、网络训练成功率等多方面均优于传统训练算法,如经典的BP算法。

数值试验也表明了这一新算法的有效性。

本文通过典型的BP算法与新算法的比较,得到了二者之间相互关系的初步结论。

从理论上证明了当惩罚因子趋于正无穷大时新算法就是BP算法,并且用数值试验说明了惩罚因子在网络训练算法中的作用和意义。

对于三层前馈神经网络来说,惩罚因子较小时,隐层神经元局部梯度的可变范围大,有利于连接权值的更新;惩罚因子较大时,隐层神经元局部梯度的可变范围小,不利于连接权值的更新,但能提高网络训练精度。

这说明了在网络训练过程中惩罚因子为何从小到大变化的原因,也说明了新算法的可行性而BP算法则时有无法更新连接权值的重大缺陷。

矿体预测在矿床地质中占有重要地位,由于输入样本量大,用以往前馈网络算法进行矿体预测效果不佳。

本文把前馈网络新算法应用到矿体预测中,取得了良好的预期效果。

本文最后指出了新算法的优点,并指出了有待改进的地方。

关键词:前馈神经网络,凸优化理论,训练算法,矿体预测,应用 Feed forward Neural Networks Training Algorithm Based on Convex Optimization and Its Application in Deposit Forcasting JIA Wen-chen (Computer Application) Directed by YE Shi-wei Abstract The paper studies primarily the application of convex optimization theory and algorithm for feed forward works’ training and convergence performance. It reviews the history of feed forward works, points out that the training of feed forward works is essentially a non-linear problem and introduces BP algorithm, its advantages as well as disadvantages and previous improvements for it. One of the big disadvantages of BP algorithm and its improvement algorithms is: because its error target function is non-convex in the weight values between neurons in different layers and exists local minimum point, thus, if the weight values enter local minimum point in weight values space work is trained, it is difficult to skip local minimum point and reach the global minimum point (i.e. the most optimal point).If this happening, the training works will be essful. To e these essential disadvantages, the paper constructs a new error target function including restriction item ording to convex function, Fenchel inequality in the conjugate of convex function and punishment function method in restriction optimization theory. When feed forward works based on the new target function is being trained, hidden layers’ outputs are seen as optimization variables. The main characteristics of the new target function are as follows: 1.With fixed hidden layers’ outputs, the new target function is convex in connecting weight variables; with fixed connecting weight values, the new target function is convex in hidden layers’ outputs. Thus, when connecting weight values and hidden layers’ outputs are optimized alternately, the new target function is convex in them, doesn’t exist local minimum point, and the algorithm’s sensitiveness is reduced for original weight values . 2.Because the punishment factor is increased gradually, weight values ’ searching space gets much bigger, so works can be trained and the possibility of entering local minimum point can be reduced to a certain extent work training process. Using these characteristics can e efficiently in the former feed forward works’ training algorithms the big disadvantage works training enters local minimum point easily. This creats a new idea for feed forward works’ learning algorithms by using convex optimization theory works training, connecting weight variables and hidden layer outputs can be optimized alternately. The new algorithm is much better than traditional algorithms for feed forward works. The numerical experiments show that the new algorithm is essful. paring the new algorithm with the traditional ones, a primary conclusion of their relationship is reached. It is proved theoretically that when the punishment factor nears infinity, the new algorithm is BP algorithm yet. The meaning and function of the punishment factor are also explained by numerical experiments. For three-layer feed forward works, when the punishment factor is smaller, hidden layer outputs’ variable range is bigger and this is in favor to updating of the connecting weights values, when the punishment factor is bigger, hidden layer outputs’ variable range is smaller and this is not in favor to updating of the connecting weights values but it can improve precision works. This explains the reason that the punishment factor should be increased gradually works training process. It also explains feasibility of the new algorithm and BP algorithm’s disadvantage that connecting weigh values can not be updated sometimes. Deposit forecasting is very important in deposit geology. The previous algorithms’ effect is not good in deposit forecasting because of much more input samples. The paper applies the new algorithm to deposit forecasting and expectant result is reached. The paper points out the new algorithm’s strongpoint as well as to-be-improved places in the end. Keywords: feed forward works, convex optimization theory, training algorithm, deposit forecasting, application 传统的BP算法及其改进算法的一个很大缺点是:由于其误差目标函数对于待学习的连接权值来说非凸的,存在局部最小点,对网络进行训练时,这些算法的权值一旦落入权值空间的局部最小点就很难跳出,因而无法达到全局最小点(即最优点)而使得网络训练失败。

针对这些缺陷,根据凸函数及其共轭的性质,利用Fenchel不等式,使用约束优化理论中的罚函数方法构造出了带有惩罚项的新误差目标函数。

用新的目标函数对前馈神经网络进行优化训练时,隐层输出也作为被优化变量。

这个目标函数的主要特点有: 1.固定隐层输出,该目标函数对连接权值来说是凸的;固定连接权值,对隐层输出来说是凸的。

这样在对连接权值和隐层输出进行交替优化时,它们所面对的目标函数都是凸函数,不存在局部最小的问题,算法对于初始权值的敏感性降低; 2.由于惩罚因子是逐渐增大的,使得权值的搜索空间变得比较大,从而对于大规模的网络也能够训练,在一定程度上降低了训练过程陷入局部最小的可能性。

这些特性能够在很大程度上有效地克服以往前馈网络的训练算法易于陷入局部最小而使网络训练失败的重大缺陷,也为利用凸优化理论研究前馈神经网络的学习算法开创了一个新思路。

在网络训练时,可以对连接权值和隐层输出进行交替优化。

把这种新算法应用到前馈神经网络训练学习中,在学习速度、泛化能力、网络训练成功率等多方面均优于传统训练算法,如经典的BP算法。

数值试验也表明了这一新算法的有效性。

本文通过典型的BP算法与新算法的比较,得到了二者之间相互关系的初步结论。

从理论上证明了当惩罚因子趋于正无穷大时新算法就是BP算法,并且用数值试验说明了惩罚因子在网络训练算法中的作用和意义。

对于三层前馈神经网络来说,惩罚因子较小时,隐层神经元局部梯度的可变范围大,有利于连接权值的更新;惩罚因子较大时,隐层神经元局部梯度的可变范围小,不利于连接权值的更新,但能提高网络训练精度。

这说明了在网络训练过程中惩罚因子为何从小到大变化的原因,也说明了新算法的可行性而BP算法则时有无法更新连接权值的重大缺陷。

矿体预测在矿床地质中占有重要地位,由于输入样本量大,用以往前馈网络算法进行矿体预测效果不佳。

本文把前馈网络新算法应用到矿体预测中,取得了良好的预期效果。

本文最后指出了新算法的优点,并指出了有待改进的地方。

关键词:前馈神经网络,凸优化理论,训练算法,矿体预测,应用 Feed forward Neural Networks Training Algorithm Based on Convex Optimization and Its Application in Deposit Forcasting JIA Wen-chen (Computer Application) Directed by YE Shi-wei Abstract The paper studies primarily the application of convex optimization theory and algorithm for feed forward works’ training and convergence performance. It reviews the history of feed forward works, points out that the training of feed forward works is essentially a non-linear problem and introduces BP algorithm, its advantages as well as disadvantages and previous improvements for it. One of the big disadvantages of BP algorithm and its improvement algorithms is: because its error target function is non-convex in the weight values between neurons in different layers and exists local minimum point, thus, if the weight values enter local minimum point in weight values space work is trained, it is difficult to skip local minimum point and reach the global minimum point (i.e. the most optimal point).If this happening, the training works will be essful. To e these essential disadvantages, the paper constructs a new error target function including restriction item ording to convex function, Fenchel inequality in the conjugate of convex function and punishment function method in restriction optimization theory. When feed forward works based on the new target function is being trained, hidden layers’ outputs are seen as optimization variables. The main characteristics of the new target function are as follows: 1.With fixed hidden layers’ outputs, the new target function is convex in connecting weight variables; with fixed connecting weight values, the new target function is convex in hidden layers’ outputs. Thus, when connecting weight values and hidden layers’ outputs are optimized alternately, the new target function is convex in them, doesn’t exist local minimum point, and the algorithm’s sensitiveness is reduced for original weight values . 2.Because the punishment factor is increased gradually, weight values ’ searching space gets much bigger, so works can be trained and the possibility of entering local minimum point can be reduced to a certain extent work training process. Using these characteristics can e efficiently in the former feed forward works’ training algorithms the big disadvantage works training enters local minimum point easily. This creats a new idea for feed forward works’ learning algorithms by using convex optimization theory works training, connecting weight variables and hidden layer outputs can be optimized alternately. The new algorithm is much better than traditional algorithms for feed forward works. The numerical experiments show that the new algorithm is essful. paring the new algorithm with the traditional ones, a primary conclusion of their relationship is reached. It is proved theoretically that when the punishment factor nears infinity, the new algorithm is BP algorithm yet. The meaning and function of the punishment factor are also explained by numerical experiments. For three-layer feed forward works, when the punishment factor is smaller, hidden layer outputs’ variable range is bigger and this is in favor to updating of the connecting weights values, when the punishment factor is bigger, hidden layer outputs’ variable range is smaller and this is not in favor to updating of the connecting weights values but it can improve precision works. This explains the reason that the punishment factor should be increased gradually works training process. It also explains feasibility of the new algorithm and BP algorithm’s disadvantage that connecting weigh values can not be updated sometimes. Deposit forecasting is very important in deposit geology. The previous algorithms’ effect is not good in deposit forecasting because of much more input samples. The paper applies the new algorithm to deposit forecasting and expectant result is reached. The paper points out the new algorithm’s strongpoint as well as to-be-improved places in the end. Keywords: feed forward works, convex optimization theory, training algorithm, deposit forecasting, application BP算法及其改进 2.1 BP算法步骤 1°随机抽取初始权值ω0; 2°输入学习样本对(Xp,Yp),学习速率η,误差水平ε; 3°依次计算各层结点输出opi,opj,opk; 4°修正权值ωk+1=ωk+ηpk,其中pk=,ωk为第k次迭代权变量; 5°若误差E<ε停止,否则转3°。

2.2 最优步长ηk的确定 在上面的算法中,学习速率η实质上是一个沿负梯度方向的步长因子,在每一次迭代中如何确定一个最优步长ηk,使其误差值下降最快,则是典型的一维搜索问题,即E(ωk+ηkpk)=(ωk+ηpk)。

令Φ(η)=E(ωk+ηpk),则Φ′(η)=dE(ωk+ηpk)/dη=E(ωk+ηpk)Tpk。

若ηk为(η)的极小值点,则Φ′(ηk)=0,即E(ωk+ηpk)Tpk=-pTk+1pk=0。

确定ηk的算法步骤如下 1°给定η0=0,h=0.01,ε0=0.00001; 2°计算Φ′(η0),若Φ′(η0)=0,则令ηk=η0,停止计算; 3°令h=2h, η1=η0+h; 4°计算Φ′(η1),若Φ′(η1)=0,则令ηk=η1,停止计算; 若Φ′(η1)>0,则令a=η0,b=η1;若Φ′(η1)<0,则令η0=η1,转3°; 5°计算Φ′(a),若Φ′(a)=0,则ηk=a,停止计算; 6°计算Φ′(b),若Φ′(b)=0,则ηk=b,停止计算; 7°计算Φ′(a+b/2),若Φ′(a+b/2)=0,则ηk=a+b/2,停止计算; 若Φ′(a+b/2)<0,则令a=a+b/2;若Φ′(a+b/2)>0,则令b=a+b/2 8°若|a-b|<ε0,则令,ηk=a+b/2,停止计算,否则转7°。

2.3 改进BP算法的特点分析 在上述改进的BP算法中,对学习速率η的选取不再由用户自己确定,而是在每次迭代过程中让计算机自动寻找最优步长ηk。

而确定ηk的算法中,首先给定η0=0,由定义Φ(η)=E(ωk+ηpk)知,Φ′(η)=dE(ωk+ηpk)/dη=E(ωk+ηpk)Tpk,即Φ′(η0)=-pTkpk≤0。

若Φ′(η0)=0,则表明此时下降方向pk为零向量,也即已达到局部极值点,否则必有Φ′(η0)<0,而对于一维函数Φ(η)的性质可知,Φ′(η0)<0则在η0=0的局部范围内函数为减函数。

故在每一次迭代过程中给η0赋初值0是合理的。

改进后的BP算法与原BP算法相比有两处变化,即步骤2°中不需给定学习速率η的值;另外在每一次修正权值之前,即步骤4°前已计算出最优步长ηk。

学习神经网络算法,看什么书比较好, 求推荐~

理论方面用: <遗传算法---理论,应用与软件实现> 王小平 西安交大出版社 (本人从十本相关书籍中选的一本) 软件实现用: <MATLAB 遗传算法工具箱及应用...

BP算法的BP算法基本介绍

含有隐层的多层前馈网络能大大提高神经网络的分类能力,但长期以来没有提出解决权值调整问题的有效算法。

1986年,Rumelhart和McCelland领导的科学家小组在《Parallel Distributed Processing》一书中,对具有非线性连续转移函数的多层前馈网络的误差反向传播(Error Back Propagation,简称BP)算法进行了详尽的分析,实现了Minsky关于多层网络的设想。

BP算法的基本思想是,学习过程由信号的正向传播与误差的反向传播两个过程组成。

正向传播时,输入样本从输入层传入,经各隐层逐层处理后,传向输出层。

若输出层的实际输出与期望的输出(教师信号)不符,则转入误差的反向传播阶段。

误差反传是将输出误差以某种形式通过隐层向输入层逐层反传,并将误差分摊给各层的所有单元,从而获得各层单元的误差信号,此误差信号即作为修正各单元权值的依据。

这种信号正向传播与误差反向传播的各层权值调整过程,是周而复始地进行的。

权值不断调整的过程,也就是网络的学习训练过程。

此过程一直进行到网络输出的误差减少到可接受的程度,或进行到预先设定的学习次数为止。

BP算法的网络结构示意图

  • 神经算法BP算法的BP算法基本介绍相关文档

Megalayer优化带宽和VPS主机主机方案策略 15M CN2优化带宽和30M全向带宽

Megalayer 商家主营业务是以独立服务器和站群服务器的,后来也陆续的有新增香港、菲律宾数据中心的VPS主机产品。由于其线路的丰富,还是深受一些用户喜欢的,有CN2优化直连线路,有全向国际线路,以及针对欧美的国际线路。这次有看到商家也有新增美国机房的VPS主机,也有包括15M带宽CN2优化带宽以及30M带宽的全向线路。Megalayer 商家提供的美国机房VPS产品,提供的配置方案也是比较多,...

HostYun 新增美国三网CN2 GIA VPS主机 采用美国原生IP低至月15元

在之前几个月中也有陆续提到两次HostYun主机商,这个商家前身是我们可能有些网友熟悉的主机分享团队的,后来改名称的。目前这个品牌主营低价便宜VPS主机,这次有可以看到推出廉价版本的美国CN2 GIA VPS主机,月费地址15元,适合有需要入门级且需要便宜的用户。第一、廉价版美国CN2 GIA VPS主机方案我们可看到这个类型的VPS目前三网都走CN2 GIA网络,而且是原生IP。根据信息可能后续...

腾讯云轻量服务器老用户续费优惠和老用户复购活动

继阿里云服务商推出轻量服务器后,腾讯云这两年对于轻量服务器的推广力度还是比较大的。实际上对于我们大部分网友用户来说,轻量服务器对于我们网站和一般的业务来说是绝对够用的。反而有些时候轻量服务器的带宽比CVM云服务器够大,配置也够好,更有是价格也便宜,所以对于初期的网站业务来说轻量服务器是够用的。这几天UCLOUD优刻得香港服务器稳定性不佳,于是有网友也在考虑搬迁到腾讯云服务器商家,对于轻量服务器官方...

神经算法为你推荐
mdm.mdm是什么扩展名?fclose为什么fclose之后remove还是不成功oracle索引如何在ORACLE数据库的字段上建立索引数秦科技数秦科技旗下有哪些区块链项目?问卷星登陆请问问卷星怎么设置答题时间?easeljs如何使用createjs来编写html5游戏基础设施即服务城市基础设施、公共服务设施、市政设施有什么区别?基础设施即服务基础设施、 产品服务、 财务和 () 这几个问题是商业模式设计需要去主要解决的。桌面管理系统最好用的电脑桌面管理软件有哪些?index是什么意思index.jsp是什么意思啊?index是什么意思?我没有英语词典
国外网站空间 vir windows主机 z.com 鲨鱼机 免费名片模板 轻博 警告本网站 免费网络电视 京东商城0元抢购 gspeed 怎样建立邮箱 qq云端 重庆双线服务器托管 个人免费主页 域名与空间 阿里云免费邮箱 秒杀品 中国电信测速网站 永久免费空间 更多