Analysis and Design of Echo State Networks

• Published in: Neural computation Vol. 19; no. 1; pp. 111 - 138
• Main Authors: Ozturk, Mustafa C, Xu, Dongming, Príncipe, José C
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.01.2007; MIT Press Journals, The
• Subjects: Applied sciences; Approximation; Artificial intelligence; Biological and medical sciences; Combinatorics; Combinatorics. Ordered structures; Computer science; control theory; systems; Connectionism. Neural networks; Design of experiments; Designs and configurations; Eigenvalues; Entropy; Exact sciences and technology; Fundamental and applied biological sciences. Psychology; General aspects. Models. Methods; Learning and adaptive systems; Linear Models; Mathematics; Neural networks; Neural Networks (Computer); Sciences and techniques of general use; Vertebrates: nervous system and sense organs; Neural computation; Error estimation; Input signal; Dynamic theory; Uniform design; Bias; Spectral radius; Eigenvalue; Entropy; Systems theory; Neural network; Function approximation; Modeling; Design; Uniform distribution; Dynamics; Echo state network; Approximation error; Frequency spectrum; Information theory
• ISSN: 0899-7667; 1530-888X
• DOI: 10.1162/neco.2007.19.1.111

The design of echo state network (ESN) parameters relies on the selection of the maximum eigenvalue of the linearized system around zero (spectral radius). However, this procedure does not quantify in a systematic manner the performance of the ESN in terms of approximation error. This article presents a functional space approximation framework to better understand the operation of ESNs and proposes an information-theoretic metric, the average entropy of echo states, to assess the richness of the ESN dynamics. Furthermore, it provides an interpretation of the ESN dynamics rooted in system theory as families of coupled linearized systems whose poles move according to the input signal dynamics. With this interpretation, a design methodology for functional approximation is put forward where ESNs are designed with uniform pole distributions covering the frequency spectrum to abide by the richness metric, irrespective of the spectral radius. A single bias parameter at the ESN input, adapted with the modeling error, configures the ESN spectral radius to the input-output joint space. Function approximation examples compare the proposed design methodology versus the conventional design.