On Connections Between Regularizations for Improving DNN Robustness

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 43; no. 12; pp. 4469 - 4476
• Main Authors: Guo, Yiwen, Chen, Long, Chen, Yurong, Zhang, Changshui
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: adversarial robustness; Artificial neural networks; Computational modeling; Deep neural networks; Image classification; Jacobian matrices; Machine learning; network property; Neural networks; Perturbation methods; Regularization; regularizations; Robustness; Task analysis; Training
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2020.3006917

This paper analyzes regularization terms proposed recently for improving the adversarial robustness of deep neural networks (DNNs), from a theoretical point of view. Specifically, we study possible connections between several effective methods, including input-gradient regularization, Jacobian regularization, curvature regularization, and a cross-Lipschitz functional. We investigate them on DNNs with general rectified linear activations, which constitute one of the most prevalent families of models for image classification and a host of other machine learning applications. We shed light on essential ingredients of these regularizations and re-interpret their functionality. Through the lens of our study, more principled and efficient regularizations can possibly be invented in the near future.