Variance Suppression: Balanced Training Process in Deep Learning

Stochastic gradient descent updates parameters with summation gradient computed from a random data batch. This summation will lead to unbalanced training process if the data we obtained is unbalanced. To address this issue, this paper takes the error variance and error mean both into consideration....

Full description

Saved in:
Bibliographic Details
Published inJournal of physics. Conference series Vol. 1207; no. 1; pp. 12013 - 12019
Main Authors Yi, T, Wang, X
Format Journal Article
LanguageEnglish
Published Bristol IOP Publishing 01.04.2019
Subjects
Algorithms
Deep learning
Errors
Machine learning
Training
Variance
Online AccessGet full text

Cover

More Information
Summary:Stochastic gradient descent updates parameters with summation gradient computed from a random data batch. This summation will lead to unbalanced training process if the data we obtained is unbalanced. To address this issue, this paper takes the error variance and error mean both into consideration. The adaptively adjusting approach of two terms trading off is also given in our algorithm. Due to this algorithm can suppress error variance, we named it Variance Suppression Gradient Descent (VSSGD). Experimental results have demonstrated that VSSGD can accelerate the training process, effectively prevent overfitting, improve the networks learning capacity from small samples.
ISSN:1742-6588
1742-6596
DOI:10.1088/1742-6596/1207/1/012013