Dual extended Kalman filtering in recurrent neural networks

• Published in: Neural networks Vol. 16; no. 2; pp. 223 - 239
• Main Authors: Leung, Chi-Sing, Chan, Lai-Wan
• Format: Journal Article
• Language: English
• Published: Oxford Elsevier Ltd 01.03.2003; Elsevier Science
• Subjects: Algorithms; Applied sciences; Detection, estimation, filtering, equalization, prediction; Electric, optical and optoelectronic circuits; Electronics; Exact sciences and technology; Information, signal and communications theory; Kalman filtering; Learning - physiology; Neural networks; Neural Networks (Computer); Recurrent neural networks; Recursive least square; Signal and communications theory; Signal, noise; Telecommunications and information theory; Recursive least square; Recurrent neural networks; Kalman filtering; Waveform; Stochastic model; Parameter estimation; Error analysis; Covariance matrix; Recursive algorithm; Computational complexity; Experimental result; System performance; Least squares method; Weight function; Deterministic model; Numerical simulation; State estimation
• ISSN: 0893-6080; 1879-2782
• DOI: 10.1016/S0893-6080(02)00230-7

In the classical deterministic Elman model, the estimation of parameters must be very accurate. Otherwise, the system performance is very poor. To improve the system performance, we can use a Kalman filtering algorithm to guide the operation of a trained recurrent neural network (RNN). In this case, during training, we need to estimate the state of hidden layer, as well as the weights of the RNN. This paper discusses how to use the dual extended Kalman filtering (DEKF) for this dual estimation and how to use our proposing DEKF for removing some unimportant weights from a trained RNN. In our approach, one Kalman algorithm is used for estimating the state of the hidden layer, and one recursive least square (RLS) algorithm is used for estimating the weights. After training, we use the error covariance matrix of the RLS algorithm to remove unimportant weights. Simulation showed that our approach is an effective joint-learning–pruning method for RNNs under the online operation.