DSTP-RNN: A dual-stage two-phase attention-based recurrent neural network for long-term and multivariate time series prediction

• Published in: Expert systems with applications Vol. 143; p. 113082
• Main Authors: Liu, Yeqi, Gong, Chuanyang, Yang, Ling, Chen, Yingyi
• Format: Journal Article
• Language: English
• Published: New York Elsevier Ltd 01.04.2020; Elsevier BV
• Subjects: Attention mechanism; Deep attention network; Dependence; Dual-stage two-phase model; Machine learning; Multivariate analysis; Neural networks; Recurrent neural networks; Spatio-temporal relationship; Time series; Time series prediction; Weight; Attention mechanism; Deep attention network; Time series prediction; Spatio-temporal relationship; Dual-stage two-phase model
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2019.113082

•We propose DSTP-RNN and DSTP-RNN-Ⅱ for long-term time series prediction.•We enhance the attention to spatio-temporal relationships of time series.•We study the deep spatial attention mechanism and give the interpretation.•Our methods outperform nine baseline methods on four datasets. Long-term prediction of multivariate time series is still an important but challenging problem. The key to solve this problem is capturing (1) the spatial correlations at the same time, (2) the spatio-temporal relationships at different times, and (3) long-term dependency of the temporal relationships between different series. Attention-based recurrent neural networks (RNN) can effectively represent and learn the dynamic spatio-temporal relationships between exogenous series and target series, but they only perform well in one-step time prediction and short-term time prediction. In this paper, inspired by human attention mechanism including the dual-stage two-phase (DSTP) model and the influence mechanism of target information and non-target information, we propose DSTP-based RNN (DSTP-RNN) and DSTP-RNN-Ⅱ respectively for long-term time series prediction. Specifically, we first propose the DSTP-based structure to enhance the spatial correlations between exogenous series. The first phase produces violent but decentralized response weight, while the second phase leads to stationary and concentrated response weight. Then, we employ multiple attentions on target series to boost the long-term dependency. Finally, we study the performance of deep spatial attention mechanism and provide interpretation. Experimental results demonstrate that the present work can be successfully used to develop expert or intelligent systems for a wide range of applications, with state-of-the-art performances superior to nine baseline methods on four datasets in the fields of energy, finance, environment and medicine, respectively. Overall, the present work carries a significant value not merely in the domain of machine intelligence and deep learning, but also in the fields of many applications.