On a continuous time model of gradient descent dynamics and instability in deep learning

• Main Authors: Rosca, Mihaela, Wu, Yan, Qin, Chongli, Dherin, Benoit
• Format: Journal Article
• Language: English
• Published: 03.02.2023
• Subjects: Computer Science - Learning; Statistics - Machine Learning

The recipe behind the success of deep learning has been the combination of neural networks and gradient-based optimization. Understanding the behavior of gradient descent however, and particularly its instability, has lagged behind its empirical success. To add to the theoretical tools available to...