Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
In this paper, we first present an explanation regarding the common occurrence of spikes in the training loss when neural networks are trained with stochastic gradient descent (SGD). We provide evidence that the spikes in the training loss of SGD are "catapults", an optimization phenomenon...
Saved in:
Main Authors | , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
07.06.2023
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | In this paper, we first present an explanation regarding the common
occurrence of spikes in the training loss when neural networks are trained with
stochastic gradient descent (SGD). We provide evidence that the spikes in the
training loss of SGD are "catapults", an optimization phenomenon originally
observed in GD with large learning rates in [Lewkowycz et al. 2020]. We
empirically show that these catapults occur in a low-dimensional subspace
spanned by the top eigenvectors of the tangent kernel, for both GD and SGD.
Second, we posit an explanation for how catapults lead to better generalization
by demonstrating that catapults promote feature learning by increasing
alignment with the Average Gradient Outer Product (AGOP) of the true predictor.
Furthermore, we demonstrate that a smaller batch size in SGD induces a larger
number of catapults, thereby improving AGOP alignment and test performance. |
---|---|
DOI: | 10.48550/arxiv.2306.04815 |