Deep Attention Network for Egocentric Action Recognition

• Published in: IEEE transactions on image processing Vol. 28; no. 8; pp. 3703 - 3713
• Main Authors: Lu, Minlong, Li, Ze-Nian, Wang, Yueming, Pan, Gang
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.08.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: action recognition; Artificial neural networks; Attention - physiology; Attention network; Cameras; Data models; Databases, Factual; egocentric vision; Head; Human Activities - classification; Human behavior; Humans; Image Processing, Computer-Assisted - methods; LSTM; Moving object recognition; Neural Networks, Computer; Pattern Recognition, Automated - methods; Predictive models; Recognition; Streaming media; Task analysis; top-down attention; Videos
• ISSN: 1057-7149; 1941-0042; 1941-0042
• DOI: 10.1109/TIP.2019.2901707

Recognizing a camera wearer's actions from videos captured by an egocentric camera is a challenging task. In this paper, we employ a two-stream deep neural network composed of an appearance-based stream and a motion-based stream to recognize egocentric actions. Based on the insight that human action and gaze behavior are highly coordinated in object manipulation tasks, we propose a spatial attention network to predict human gaze in the form of attention map. The attention map helps each of the two streams to focus on the most relevant spatial region of the video frames to predict actions. To better model the temporal structure of the videos, a temporal network is proposed. The temporal network incorporates bi-directional long short-term memory to model the long-range dependencies to recognize egocentric actions. The experimental results demonstrate that our method is able to predict attention maps that are consistent with human attention and achieve competitive action recognition performance with the state-of-the-art methods on the GTEA Gaze and GTEA Gaze+ datasets.