Recurrent attention unit: A new gated recurrent unit for long-term memory of important parts in sequential data

• Published in: Neurocomputing (Amsterdam) Vol. 517; pp. 1 - 9
• Main Authors: Niu, Zhaoyang, Zhong, Guoqiang, Yue, Guohua, Wang, Li-Na, Yu, Hui, Ling, Xiao, Dong, Junyu
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 14.01.2023
• Subjects: Attention mechanism; Gated recurrent unit (GRU); Memory; Recurrent attention unit (RAU); Recurrent neural networks (RNNs); Attention mechanism; Recurrent attention unit (RAU); Gated recurrent unit (GRU); Recurrent neural networks (RNNs); Memory
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2022.10.050

Gated recurrent unit (GRU) is a variant of the recurrent neural network (RNN). It has been widely used in many applications, such as handwriting recognition and natural language processing. However, GRU can only memorize the sequential information, but lacks the capability of adaptively paying attention to important parts in the sequences. In this paper, we propose a novel RNN model, called recurrent attention unit (RAU), which can seamlessly integrate the attention mechanism into the interior of the GRU cell by adding an attention gate. The attention gate enhances the ability of RAU to remember long-term information and pay attention to important parts in the sequential data. Extensive experiments on adding problem, image classification, sentiment classification and language modeling show that RAU consistently outperforms GRU and other related models.