Deep reservoir computing: A critical experimental analysis

• Published in: Neurocomputing (Amsterdam) Vol. 268; pp. 87 - 99
• Main Authors: Gallicchio, Claudio, Micheli, Alessio, Pedrelli, Luca
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 13.12.2017
• Subjects: Deep Learning; Deep neural networks; Echo State Networks; Multiple time-scale dynamics; Recurrent neural networks; Reservoir computing; Recurrent neural networks; Deep neural networks; Reservoir computing; Echo State Networks; Deep Learning; Multiple time-scale dynamics
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2016.12.089

In this paper, we propose an empirical analysis of deep recurrent neural network (RNN) architectures with stacked layers. The main aim is to address some fundamental open research issues on the significance of creating deep layered architectures in RNN and to characterize the inherent hierarchical representation of time in such models, especially for efficient implementations. In particular, the analysis aims at the study and proposal of approaches to develop and enhance hierarchical dynamics in deep architectures within the efficient Reservoir Computing (RC) framework for RNN modeling. The effect of a deep layered organization of RC models is investigated in terms of both occurrence of multiple time-scale and increasing of richness of the dynamics. It turns out that a deep layering of recurrent models allows an effective diversification of temporal representations in the layers of the hierarchy, by amplifying the effects of the factors influencing the time-scales and the richness of the dynamics, measured as the entropy of recurrent units activations. The advantages of the proposed approach are also highlighted by measuring the increment of the short-term memory capacity of the RC models.