The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies

We study the relationship between the frequency of a function and the speed at which a neural network learns it. We build on recent results that show that the dynamics of overparameterized neural networks trained with gradient descent can be well approximated by a linear system. When normalized trai...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Basri, Ronen, Jacobs, David, Kasten, Yoni, Kritchman, Shira
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 02.12.2019
Subjects
Bias
Eigenvalues
Eigenvectors
Harmonic functions
Hyperspheres
Neural networks
Spherical harmonics
Online AccessGet full text

Cover

More Information
Summary:We study the relationship between the frequency of a function and the speed at which a neural network learns it. We build on recent results that show that the dynamics of overparameterized neural networks trained with gradient descent can be well approximated by a linear system. When normalized training data is uniformly distributed on a hypersphere, the eigenfunctions of this linear system are spherical harmonic functions. We derive the corresponding eigenvalues for each frequency after introducing a bias term in the model. This bias term had been omitted from the linear network model without significantly affecting previous theoretical results. However, we show theoretically and experimentally that a shallow neural network without bias cannot represent or learn simple, low frequency functions with odd frequencies. Our results lead to specific predictions of the time it will take a network to learn functions of varying frequency. These predictions match the empirical behavior of both shallow and deep networks.
ISSN:2331-8422