On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures

• Published in: IEEE transaction on neural networks and learning systems Vol. 25; no. 8; pp. 1553 - 1565
• Main Authors: Bianchini, Monica, Scarselli, Franco
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.08.2014; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Activation; Activation analysis; Algorithms; Applied sciences; Architecture; Artificial intelligence; Betti numbers; Biological neural networks; Classification; Complexity; Complexity theory; Computer architecture; Computer science; control theory; systems; Computer Simulation; Connectionism. Neural networks; Data processing. List processing. Character string processing; deep neural networks; Exact sciences and technology; function approximation; Heuristic; Learning; Mathematical models; Memory organisation. Data processing; Models, Theoretical; Nerve Net; Neural networks; Neurons; Pattern Recognition, Automated - methods; Pattern recognition. Digital image processing. Computational geometry; Polynomials; Software; topological complexity; Upper bound; Vapnik > Chervonenkis dimension (VC-dim); Vapnik–Chervonenkis dimension (VC-dim); topological complexity; function approximation; deep neural networks; Betti numbers; Lower bound; Automatic classification; Great depth; Vapnik-Chervonenkis dimension (VC-dim); Neural network; Topology; Function approximation; Deep learning; Upper bound; Experimental result; Feedforward neural nets; Activation function; Heuristic method; Probability learning; Sigmoidal transformation; Feedforward
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2013.2293637

Recently, researchers in the artificial neural network field have focused their attention on connectionist models composed by several hidden layers. In fact, experimental results and heuristic considerations suggest that deep architectures are more suitable than shallow ones for modern applications, facing very complex problems, e.g., vision and human language understanding. However, the actual theoretical results supporting such a claim are still few and incomplete. In this paper, we propose a new approach to study how the depth of feedforward neural networks impacts on their ability in implementing high complexity functions. First, a new measure based on topological concepts is introduced, aimed at evaluating the complexity of the function implemented by a neural network, used for classification purposes. Then, deep and shallow neural architectures with common sigmoidal activation functions are compared, by deriving upper and lower bounds on their complexity, and studying how the complexity depends on the number of hidden units and the used activation function. The obtained results seem to support the idea that deep networks actually implements functions of higher complexity, so that they are able, with the same number of resources, to address more difficult problems.