前向人工神经网络敏感性研究课件

《前向人工神经网络敏感性研究课件》由会员分享，可在线阅读，更多相关《前向人工神经网络敏感性研究课件（38页珍藏版）》请在装配图网上搜索。

1、前向人工神经网络敏感性研究1前向人工神经网络敏感性研究2003年10月 前向人工神经网络敏感性研究2一一. . 引言引言 1. 前向神经网络前向神经网络( (FNN)FNN)介绍介绍 神经元 离散型：自适应线性元（Adaline） 连续型：感知机（Perceptron） 神经网络 离散型：多层自适应线性网（Madaline） 连续型：多层感知机（BP网或MLP）前向人工神经网络敏感性研究3问题问题 硬件精度对权的影响 环境噪音对输入的影响 动机动机 参数的扰动对网络会产生怎样影响？ 如何衡量网络输出偏差的大小？2. 研究提出研究提出前向人工神经网络敏感性研究4建立网络输出与网络参数扰动之间的关

2、系分析该关系，揭示网络的行为规律量化网络输出偏差3. 研究内容研究内容),(WXSY),(WXSY),(WXSY前向人工神经网络敏感性研究5指导网络设计，增强网络抗干扰能力度量网络性能，如容错和泛化能力研究其它网络课题的基础，如网络结构的 裁剪和参数的挑选等4. 研究意义研究意义前向人工神经网络敏感性研究6 Madaline的敏感性n维几何模型（超球面） M. Stevenson, R. Winter, and B. Widrow, “Sensitivity of Feedforward Neural Networks to Weight Errors,” IEEE Trans. on Neu

3、ral, Networks, vol. 1, no. 1, 1990. 统计模型（方差） S. W. Pich, “The Selection of Weight Accuracies for Madalines,” IEEE Trans. on Neural Networks, vol. 6, no. 2, 1995.（典型（典型方法和方法和）前向人工神经网络敏感性研究7分析方法（偏微分） S. Hashem, “Sensitivity Analysis for Feed- Forward Artificial Neural Networks with Differentiable Acti

4、vation Functions”, Proc. of IJCNN, vol. 1, 1992. 统计方法（标准差） J. Y. Choi & C. H. Choi, “Sensitivity Ana- lysis of Multilayer Perceptron with Differ- entiable Activation Functions,” IEEE Trans. on Neural Networks, vol. 3, no. 1, 1992.2. MLP的敏感性前向人工神经网络敏感性研究8输入属性筛选 J. M. Zurada, A. Malinowski, S. Usui, “

5、Perturbation Method for Deleting Redundant Inputs of Perceptron Networks”, Neurocomputing, vol. 14, 1997. 网络结构裁减 A. P. Engelbrecht, “A New Pruning Heuristic Based on Variance Analysis of Sensitivity Information”, IEEE Trans. on Neural Networks, vol. 12, no. 6, 2001.3. 敏感性的应用前向人工神经网络敏感性研究9 J.L. Berni

6、er et al, “A Quantitive Study of Fault Tolerance, Noise Immunity and Generalization Ability of MLPs,” Neural Computation, vol. 12, 2000. 容错和泛化问题前向人工神经网络敏感性研究10三三. . 研究方法研究方法1. 自底向上自底向上方法方法单个神经元整个网络2. 概率统计方法概率统计方法概率率（离散型）均值（连续型）3 3. n-维几何模型维几何模型超矩形的顶点（离散型）超矩形体（连续型）前向人工神经网络敏感性研究11四四. .已获成果（代表性论文）已获成果（

7、代表性论文） 敏感性分析： “Sensitivity Analysis of Multilayer Percep- tron to Input and Weight Perturbations,” IEEE Trans. on Neural Networks, vol. 12, no.6, pp. 1358-1366, Nov. 2001. 前向人工神经网络敏感性研究12 敏感性量化： “A Quantified Sensitivity Measure for Multi- layer Perceptron to Input Perturbation,” Neural Computation,

8、 vol. 15, no. 1, pp. 183-212, Jan. 2003.前向人工神经网络敏感性研究13隐层节点的裁剪（敏感性应用）： “Hidden Neuron Pruning for Multilayer Perceptrons Using Sensitivity Measure,” Proc. of IEEE ICMLC2002, pp. 1751-1757, Nov. 2002. 输入属性重要性的判定（敏感性应用）： “Determining the Relevance of Input Features for Multilayer Perceptrons,” Proc. o

9、f IEEE SMC2003, Oct. 2003.前向人工神经网络敏感性研究14五五. . 未来工作未来工作 进一步完善已有的结果进一步完善已有的结果, ,使之更加实用使之更加实用 放松限制条件 扩大分析范围 精确量化计算 进一步应用所得的结果进一步应用所得的结果, ,解决实际问题解决实际问题 探索新方法探索新方法, ,研究新类型的网络研究新类型的网络前向人工神经网络敏感性研究15结束谢谢各位！谢谢各位！ 前向人工神经网络敏感性研究16 Inputs Weights Neurons action Output 1x 1w 2x 2w . 1 0jjwx . jjwx )(jjwxf y .

10、1 0jjwx 1nw 1nx nw nx 1, 1,1, 1yRWXnn前向人工神经网络敏感性研究17 Inputs Weights Neurons action Output 1x 1w 2x 2w . . jjwx )(jjwxf y= 1/(1+jjwxe) . 1nw 1nx nw nx 1 , 0,1 , 0yRWXnn前向人工神经网络敏感性研究18 Input X1 Layer 1 . Layer (L-1) Layer L Output YL 2n 2Ln 11x 2n 2Ln Ly1 12x 110nx 10nx 2n 2Ln LnLy 2n 2Ln 0ninputs 1n

11、neurons 1Lnneurons Lnneurons前向人工神经网络敏感性研究19 WW X FNN YY 前向人工神经网络敏感性研究20 W XX FNN YY 前向人工神经网络敏感性研究21Effects of input & weight deviations on neurons sensitivitySensitivity increases with input and weigh deviations, but the increase has an upper bound.前向人工神经网络敏感性研究22Effects of input dimension on neuron

12、s sensitivityThere exists an optimal value for the dimension of input, which yields the highest sensitivity value.前向人工神经网络敏感性研究23Effects of input & weight deviations on MLPs sensitivitySensitivity of an MLP increases with the input and weight deviations. 前向人工神经网络敏感性研究24Effects of the number of neuro

13、ns in a layer Sensitivity of MLPs: n-2-2-1 | 1n 10 to the dimension of input. 前向人工神经网络敏感性研究25 Sensitivity of MLPs: 2-n-2-1 | 1n 10 to the number of neurons in the 1st layer. 前向人工神经网络敏感性研究26 Sensitivity of MLPs: 2-2-n-1 | 1n 10 to the number of neurons in the 2nd layer .There exists an optimal value

14、for the number of neurons in a layer, which yields the highest sensitivity value. The nearer a layer to the output layer is, The more effect the number of neurons in the layer has.前向人工神经网络敏感性研究27Effects of the number of layers Sensitivity of MLPs:2-1,2-2-1,.,2-2-2-2-2-2-2-2-2-2-1 to the number of la

15、yers.Sensitivity decreases with the number increasing, and the decrease almost levels off when the number becomes large.前向人工神经网络敏感性研究28Sensitivity of the neurons with 2-dimensional input 前向人工神经网络敏感性研究29Sensitivity of the neurons with 3-dimensional input 前向人工神经网络敏感性研究30Sensitivity of the neurons with

16、 4-dimensional input 前向人工神经网络敏感性研究31Sensitivity of the neurons with 5-dimensional input 前向人工神经网络敏感性研究32Sensitivity of the MLPs: 2-2-1, 2-3-1,2-2-2-1 前向人工神经网络敏感性研究33 Simulation 1 (Function Approximation) Implement an MLP to approximate the function: where Implementation considerations The MLP archite

17、cture is restricted to 2-n-1. The convergence condition is MES-goal=0.01&Epoch105. The lowest trainable number of hidden neurons is n=5. The pruning processes start with MLPs of 2-5-1 and stop at an architecture of 2-4-1. The relevant data used by and resulted from the pruning process are listed in

18、Table 1 and Table 2.21121215 .0),(xxexxxxxF 1 , 0 1 , 0),(21xx前向人工神经网络敏感性研究34TABLE 1. Data for 3 MLPs with 5 hidden neurons to realize the functionMLP2-5-1EpochMSE (training)MSE (testing)Trained weights and biasMSE-(goal=0.01 & epoch=100000)Sensitivity Relevance1305860.0009998160.0117005-12.9212 -0.

19、2999 33.7943 -34.6057 31.4768 -31.0169-0.5607 -0.8140 1.1737 -1.1026-5.4507 12.7341 -13.0816 -12.0171 8.7152 bias=00.0317940.0022720.0014060.0270660.0018150.17330.02890.01840.32530.01582652090.0009999590.0124573 32.6223 -33.3731-0.7361 0.7202-31.8412 31.2399-15.1872 -0.0937-0.3989 -1.0028 11.9959 -1

20、5.4905 12.2103 -6.0877 -12.5057 bias=00.0021760.0004630.0018210.0310170.0270680.02610.00720.02220.18880.33853260940.0009999440.0120354-15.0940 17.6184-19.9163 21.4109-14.0535 -0.8460 1.0263 -0.1258 26.7757 -26.1259 8.8172 -18.6532 -6.8307 16.8506 -10.4671 bias=00.0135470.0066610.0262200.0283520.0023

21、240.11940.12420.17910.47770.0243前向人工神经网络敏感性研究35TABLE 2. Data for the 3 pruned MLPs with 4 hidden neurons to realize the functionMLP2-4-1EpochMSE (training)MSE (testing)Retrained weights and bias(goal=0.01 & epoch=100000)SensitivityRelevance1(Obtained by removing the 5th neuron from the MLP of 2-5-1)

22、22510.0009999980.0114834-14.4387 -0.7003 34.8366 -35.6080 33.1285 -32.6271-1.5065 0.0184-5.7036 13.0579 - 1 3 . 2 4 5 7 - 1 2 . 1 8 0 3 bias=4.23490.0270140.0021000.0014600.0313430.15410.02740.01930.38182(Obtained by removing the 2nd neuron from the MLP of 2-5-1) 19450.0009999210.0119645 33.5805 -34

23、.2727-32.9313 32.3172-15.8016 -0.5610-1.3318 0.010312.6267 12.7961 -6.1782 -13.3652 bias=-7.94680.0019540.0018000.0269020.0292830.02470.02300.16620.39143(Obtained by removing the 5th neuron from the MLP of 2-5-1)132530.0009999710.011926-34.3974 33.8148-34.3250 34.7990-1.2909 0.0198 11.8097 0.8879 15

24、.7984 -15.6503 -12.9606 6.0722 bias=-1.41940.0016370.0013160.0288340.0281220.02590.02060.37370.1708前向人工神经网络敏感性研究36 Simulation 2 (Classification) Implement an MLP to solve the XOR problem: 0 1Implementation considerations The MLP architecture is restricted to 2-n-1. The convergence condition is MES-g

25、oal=0.1&Epoch105. The pruning processes start with MLPs of 2-5-1 and stop at an architecture of 2-4-1. The relevant data used by and resulted from the pruning process are listed in Table 3 and Table 4.),(21xxF15 . 0&15 . 05 . 00&5 . 002121xxorxx5 . 00&15 . 015 . 0&5 . 002121xxorxx前向人工神经网络敏感性研究37TABL

26、E 3. Data for 3 MLPs with 5 hidden neurons to realize the functionMLP2-5-1EpochMSE (training)MSE (testing)Trained weights and bias(goal=0.1 & epoch=100000)SensitivityRelevance1445180.09999970.109217 2.8188 -8.1143 2.4420 -0.5450 2.5766 3.7037 1.4955 -2.9245-2.5714 -3.7124 14.0153 -43.9907 28.0636 19

27、.5486 -68.6432 bias=00.0475990.0357470.0315180.0273550.0315130.66711.57250.88450.53482.16322510980.09999980.113006 1.4852 -3.8902 1.0692 0.1466-1.0723 -0.1455-7.0301 2.5695-3.1382 -2.8094 23.9314 -19.1824 27.1565 14.9694 -91.6363 bias=00.0375930.0201700.0201780.0455040.0325500.89970.38690.54800.6812

28、2.98283336310.09999940.11369 3.2920 2.9094-1.0067 3.4724-7.0578 2.4377-3.2921 -2.9096 1.5303 -0.0606 45.7579 -30.0598 16.5386 -52.2874 -29.7040 bias=00.0314980.0391660.0462100.0314970.0317151.44131.17730.76421.64690.9421前向人工神经网络敏感性研究38TABLE 4. Data for the 3 pruned MLPs with 4 hidden neurons to real

29、ize the functionMLP2-4-1EpochMSE (training)MSE (testing)Retrained weights and bias(goal=0.1 & epoch=100000)SensitivityRelevance1(Obtained by removing the 4th neuron from the MLP of 2-5-1)226110.09999990.109085 2.8745 -6.8849 1.9844 0.0405 2.6295 3.8648-2.6270 -3.8656 22.5649 -51.3458 3 3 . 0 9 8 2 -

30、 7 4 . 4 3 7 1 bias=5.55700.0431730.0286270.0307080.0307170.97421.46991.01642.28652(Obtained by removing the 2nd neuron from the MLP of 2-5-1)144570.09999980.112792 1.1511 -3.9352-1.4080 -0.2348-6.8277 2.3307-3.2002 -2.9670 26.3668 31.5437 16.3482 -98.8089 bias=-12.46560.0408410.0295910.0459790.0316121.07680.93340.75173.12353(Obtained by removing the 3rd neuron from the MLP of 2-5-1)175010.09999970.111499 3.0386 3.7789-1.3471 4.6670-3.0386 -3.7789 3.5143 -0.7579 59.1526 -34.0215 - 5 8 . 5 9 4 9 - 3 6 . 1 7 6 1 bias=1.74740.0291140.0430970.0291140.0413721.72221.46621.70591.4967