文章基本信息

标题：Theoretical Analysis of Learning Speed in Gradient Descent Algorithm Replacing Derivative with Constant
本地全文：下载
作者：Kazuyuki Hara ; Kentaro Katahira
期刊名称：Information and Media Technologies
电子版ISSN：1881-0896
出版年度：2014
卷号：9
期号：1
页码：61-66
DOI：10.11185/imt.9.61
出版社：Information and Media Technologies Editorial Board
摘要：In on-line gradient descent learning, the local property of the derivative term of the output function can slowly converge. Improving the derivative term, such as by using the natural gradient, has been proposed for speeding up the convergence. Beside this sophisticated method, we propose an algorithm that replaces the derivative term with a constant and show that this greatly increases convergence speed when the learning step size is less than 2.7, which is near the optimal learning step size. The proposed algorithm is inspired by linear perceptron learning and can avoid locality of the derivative term. We derived the closed deterministic differential equations by using a statistical mechanics method and show the validity of theoretical results by comparing them with computer simulation solutions. In real problems, the optimum learning step size is not given in advance. Therefore, the learning step size must be small. The proposed method is useful in this case.
关键词：learning speed;derivative;gradient descent algorithm;simple perceptron;statistical mechanics