Multi-scale aggregation network for temporal action proposals

• Published in: Pattern recognition letters Vol. 122; pp. 60 - 65
• Main Authors: Wang, Zheng, Chen, Kai, Zhang, Mingxing, He, Peilin, Wang, Yajie, Zhu, Ping, Yang, Yang
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01.05.2019; Elsevier Science Ltd
• Subjects: Agglomeration; Labeling; Proposals
• ISSN: 0167-8655; 1872-7344
• DOI: 10.1016/j.patrec.2019.02.007

•We propose a network with temporal multi-layer dilated convolution for TAP.•We propose the unit level and proposal level multi-scale aggregation strategies.•We propose to take the soft labelling to facilitate action boundary unit prediction. Temporal action detection is a very challenging and valuable task for video analysis and applications. The detection results, to a great extent, rely on the quality of temporal action proposals. However, temporal actions in videos vary dramatically, e.g. from a fraction of a second to minutes, which causes much difficulties for accurate temporal action proposals. In this paper, we propose a multi-scale aggregation network to overcome those variations for temporal action proposals. Our proposed network generates an actionness score sequence for the input video to automatically perceive the duration of actions, and thus can dynamically generate corresponding lengths of action proposals for them. For more reliable actionness prediction, we propose to adaptively explore the intrinsic short and long dependencies in action by two multi-scale aggregation strategies: unit level multi-scale aggregation and proposal level multi-scale aggregation. We also propose to take the soft labelling to facilitate the actionness prediction for the units near the action boundaries. Extensive experiments on THUMOS14 dataset have demonstrated the effectiveness of our proposed method.