Neural Network Identifiability for a Family of Sigmoidal Nonlinearities

• Published in: Constructive approximation Vol. 55; no. 1; pp. 173 - 224
• Main Authors: Vlačić, Verner, Bölcskei, Helmut
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.02.2022; Springer Nature B.V
• Subjects: Analysis; Artificial neural networks; Computer architecture; Mathematics; Mathematics and Statistics; Neural networks; Nonlinearity; Numerical Analysis; Questions; Identifiability; Sigmoidal nonlinearities; 46-XX Functional analysis; Deep neural networks; 30-XX Functions of a complex variable; 22-XX Topological groups; Lie groups; 68T07 Artificial neural networks and deep learning
• ISSN: 0176-4276; 1432-0940
• DOI: 10.1007/s00365-021-09544-3

This paper addresses the following question of neural network identifiability: Does the input–output map realized by a feed-forward neural network with respect to a given nonlinearity uniquely specify the network architecture, weights, and biases? The existing literature on the subject (Sussman in Neural Netw 5(4):589–593, 1992; Albertini et al. in Artificial neural networks for speech and vision, 1993; Fefferman in Rev Mat Iberoam 10(3):507–555, 1994) suggests that the answer should be yes, up to certain symmetries induced by the nonlinearity, and provided that the networks under consideration satisfy certain “genericity conditions.” The results in Sussman (1992) and Albertini et al. (1993) apply to networks with a single hidden layer and in Fefferman (1994) the networks need to be fully connected. In an effort to answer the identifiability question in greater generality, we derive necessary genericity conditions for the identifiability of neural networks of arbitrary depth and connectivity with an arbitrary nonlinearity. Moreover, we construct a family of nonlinearities for which these genericity conditions are minimal , i.e., both necessary and sufficient . This family is large enough to approximate many commonly encountered nonlinearities to within arbitrary precision in the uniform norm.