The Role of Regularization in Overparameterized Neural Networks

In this paper, we consider gradient descent on a regularized loss function for training an overparametrized neural network. We model the algorithm as an ODE and show how overparameterization and regularization work together to provide the right tradeoff between training and generalization errors.

Saved in:
Bibliographic Details
Published in2020 59th IEEE Conference on Decision and Control (CDC) pp. 4683 - 4688
Main Authors Satpathi, Siddhartha, Gupta, Harsh, Liang, Shiyu, Srikant, R
Format Conference Proceeding
LanguageEnglish
Published IEEE 14.12.2020
Subjects
Mathematical model
Noise measurement
Probabilistic logic
Task analysis
Training
Training data
Trajectory
Online AccessGet full text

Cover

More Information
Summary:In this paper, we consider gradient descent on a regularized loss function for training an overparametrized neural network. We model the algorithm as an ODE and show how overparameterization and regularization work together to provide the right tradeoff between training and generalization errors.
ISSN:2576-2370
DOI:10.1109/CDC42340.2020.9304386