PCG-TAL: Progressive Cross-Granularity Cooperation for Temporal Action Localization

• Published in: IEEE transactions on image processing Vol. 30; pp. 2103 - 2113
• Main Authors: Su, Rui, Xu, Dong, Sheng, Lu, Ouyang, Wanli
• Format: Journal Article
• Language: English
• Published: United States IEEE 2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Benchmark testing; Cooperation; cross-granularity cooperation; cross-stream cooperation; Feature extraction; Grammar; Kernel; Localization; Modules; Proposals; Spatiotemporal phenomena; Temporal action localization; Three-dimensional displays
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2020.3044218

There are two major lines of works, i.e. , anchor-based and frame-based approaches, in the field of temporal action localization. But each line of works is inherently limited to a certain detection granularity and cannot simultaneously achieve high recall rates with accurate action boundaries. In this work, we propose a progressive cross-granularity cooperation (PCG-TAL) framework to effectively take advantage of complementarity between the anchor-based and frame-based paradigms, as well as between two-view clues ( i.e. , appearance and motion). Specifically, our new Anchor-Frame Cooperation (AFC) module can effectively integrate both two-granularity and two-stream knowledge at the feature and proposal levels, as well as within each AFC module and across adjacent AFC modules. Specifically, the RGB-stream AFC module and the flow-stream AFC module are stacked sequentially to form a progressive localization framework. The whole framework can be learned in an end-to-end fashion, whilst the temporal action localization performance can be gradually boosted in a progressive manner. Our newly proposed framework outperforms the state-of-the-art methods on three benchmark datasets the THUMOS14, ActivityNet v1.3 and UCF-101-24, which clearly demonstrates the effectiveness of our framework.