diffGrad: An Optimization Method for Convolutional Neural Networks

• Published in: IEEE transaction on neural networks and learning systems Vol. 31; no. 11; pp. 4500 - 4511
• Main Authors: Dubey, Shiv Ram, Chakraborty, Soumendu, Roy, Swalpa Kumar, Mukherjee, Snehasis, Singh, Satish Kumar, Chaudhuri, Bidyut Baran
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.11.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptive moment estimation (Adam); Artificial neural networks; Computer vision; Convergence; difference of gradient; Estimation; Experiments; gradient descent; image classification; Machine learning; Neural networks; Optimization; Optimization methods; Optimization techniques; Parameters; residual network; Source code; Stochasticity; Training

Stochastic gradient descent (SGD) is one of the core techniques behind the success of deep neural networks. The gradient provides information on the direction in which a function has the steepest rate of change. The main problem with basic SGD is to change by equal-sized steps for all parameters, ir...