M-adapter: Multi-level image-to-video adaptation for video action recognition

• Published in: Computer vision and image understanding Vol. 249; p. 104150
• Main Authors: Li, Rongchang, Xu, Tianyang, Wu, Xiao-Jun, Li, Linze, Yang, Xiao, Shen, Zhongwei, Kittler, Josef
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.12.2024
• Subjects: Action recognition; Image-to-video adaptation; Parameter-efficient transfer learning; Action recognition; Image-to-video adaptation; 41A10; 65D05; Parameter-efficient transfer learning; 65D17; 41A05
• ISSN: 1077-3142
• DOI: 10.1016/j.cviu.2024.104150

With the growing size of visual foundation models, training video models from scratch has become costly and challenging. Recent attempts focus on transferring frozen pre-trained Image Models (PIMs) to video fields by tuning inserted learnable parameters such as adapters and prompts. However, these methods require saving PIM activations for gradient calculations, leading to limited savings of GPU memory. In this paper, we propose a novel parallel branch that adapts the multi-level outputs of the frozen PIM for action recognition. It avoids passing gradients through the PIMs, thus naturally owning much lower GPU memory footprints. The proposed adaptation branch consists of hierarchically combined multi-level output adapters (M-adapters), comprising a fusion module and a temporal module. This design digests the existing discrepancies between the pre-training task and the target task with lower training costs. We show that when using larger models or on scenarios with higher demands for temporal modelling, the proposed method performs better than those with the full-parameter tuning manner. Finally, despite only tuning fewer parameters, our method achieves superior or comparable performance against current state-of-the-art methods. •A parallel parameter-efficient tuning method with less cost than current methods.•A multi-level adapter that adapts pre-trained image models for action recognition.•Comprehensive comparisons between full-parameter and proposed tuning approach.•Competitive performance to SOTA results while requiring lower training costs.