Efficient BackProp

• Published in: Neural Networks: Tricks of the Trade pp. 9 - 48
• Main Authors: LeCun, Yann A., Bottou, Léon, Orr, Genevieve B., Müller, Klaus-Robert
• Format: Book Chapter
• Language: English
• Published: Berlin, Heidelberg Springer Berlin Heidelberg 2012
• Series: Lecture Notes in Computer Science
• Subjects: Conjugate Gradient; Gradient Descent; Handwritten Digit; Neural Information Processing System; Newton Algorithm

The convergence of back-propagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explan...