Stochastic Gradient Descent Tricks

• Published in: Neural Networks: Tricks of the Trade pp. 421 - 436
• Main Author: Bottou, Léon
• Format: Book Chapter
• Language: English
• Published: Berlin, Heidelberg Springer Berlin Heidelberg 2012
• Series: Lecture Notes in Computer Science
• Subjects: Conditional Random Field; Empirical Risk; Learning Rate; Stochastic Gradient; Support Vector Machine

Chapter 1 strongly advocates the stochastic back-propagation method to train neural networks. This is in fact an instance of a more general technique called stochastic gradient descent (SGD). This chapter provides background material, explains why SGD is a good learning algorithm when the training s...