Determination of the Edge of Criticality in Echo State Networks Through Fisher Information Maximization

• Published in: IEEE transaction on neural networks and learning systems Vol. 29; no. 3; pp. 706 - 717
• Main Authors: Livi, Lorenzo, Bianchi, Filippo Maria, Alippi, Cesare
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2018; The Institute of Electrical and Electronics Engineers, Inc. (IEEE); Institute of Electrical and Electronics Engineers (IEEE)
• Subjects: Benchmarks; Computation; Computer applications; Configuration management; Configurations; Data processing; Echo state network (ESN); edge of criticality; Fisher information; Fysikk: 430; Jacobian matrices; Learning systems; Matematikk og Naturvitenskap: 400; Mathematics and natural science: 400; Maximization; Neural networks; Neurons; nonparametric estimation; Nonparametric statistics; Parameter identification; Physics: 430; Probability density function; Probability density functions; Recurrent neural networks; Reservoirs; Short term memory; Training; VDP
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2016.2644268

It is a widely accepted fact that the computational capability of recurrent neural networks (RNNs) is maximized on the so-called "edge of criticality." Once the network operates in this configuration, it performs efficiently on a specific application both in terms of: 1) low prediction error and 2) high short-term memory capacity. Since the behavior of recurrent networks is strongly influenced by the particular input signal driving the dynamics, a universal, application-independent method for determining the edge of criticality is still missing. In this paper, we aim at addressing this issue by proposing a theoretically motivated, unsupervised method based on Fisher information for determining the edge of criticality in RNNs. It is proved that Fisher information is maximized for (finite-size) systems operating in such critical regions. However, Fisher information is notoriously difficult to compute and requires the analytic form of the probability density function ruling the system behavior. This paper takes advantage of a recently developed nonparametric estimator of the Fisher information matrix and provides a method to determine the critical region of echo state networks (ESNs), a particular class of recurrent networks. The considered control parameters, which indirectly affect the ESN performance, are explored to identify those configurations lying on the edge of criticality and, as such, maximizing Fisher information and computational performance. Experimental results on benchmarks and real-world data demonstrate the effectiveness of the proposed method.