Geometry of Optimization and Implicit Regularization in Deep Learning

We argue that the optimization plays a crucial role in generalization of deep learning models through implicit regularization. We do this by demonstrating that generalization ability is not controlled by network size but rather by some other implicit control. We then demonstrate how changing the emp...

Full description

Saved in:
Bibliographic Details
Main Authors Neyshabur, Behnam, Tomioka, Ryota, Salakhutdinov, Ruslan, Srebro, Nathan
Format Journal Article
LanguageEnglish
Published 08.05.2017
Subjects
Computer Science - Learning
Online AccessGet full text

Cover

More Information
Summary:We argue that the optimization plays a crucial role in generalization of deep learning models through implicit regularization. We do this by demonstrating that generalization ability is not controlled by network size but rather by some other implicit control. We then demonstrate how changing the empirical optimization procedure can improve generalization, even if actual optimization quality is not affected. We do so by studying the geometry of the parameter space of deep networks, and devising an optimization algorithm attuned to this geometry.
DOI:10.48550/arxiv.1705.03071