Implicit Regularization of Dropout

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 46; no. 6; pp. 4206 - 4217
• Main Authors: Zhang, Zhongwang, Xu, Zhi-Qin John
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.06.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Behavioral sciences; Complexity theory; condensation; dropout; flatness; implicit regularization; Jacobian matrices; Neural networks; Neurons; Regularization; Regularization methods; Stochastic processes; Training
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2024.3357172

It is important to understand how dropout, a popular regularization method, aids in achieving a good generalization solution during neural network training. In this work, we present a theoretical derivation of an implicit regularization of dropout, which is validated by a series of experiments. Additionally, we numerically study two implications of the implicit regularization, which intuitively rationalizes why dropout helps generalization. First, we find that input weights of hidden neurons tend to condense on isolated orientations trained with dropout. Condensation is a feature in the non-linear learning process, which makes the network less complex. Second, we find that the training with dropout leads to the neural network with a flatter minimum compared with standard gradient descent training, and the implicit regularization is the key to finding flat solutions. Although our theory mainly focuses on dropout used in the last hidden layer, our experiments apply to general dropout in training neural networks. This work points out a distinct characteristic of dropout compared with stochastic gradient descent and serves as an important basis for fully understanding dropout.