Self-Supervised Video Representation Learning with Meta-Contrastive Network

• Published in: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 8219 - 8229
• Main Authors: Lin, Yuanze, Guo, Xun, Lu, Yan
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.10.2021
• Subjects: Computer vision; Multitasking; Representation learning; Task analysis; Training; Transfer/Low-shot/Semi/Unsupervised Learning; Video analysis and understanding
• ISSN: 2380-7504
• DOI: 10.1109/ICCV48922.2021.00813

Self-supervised learning has been successfully applied to pre-train video representations, which aims at efficient adaptation from pre-training domain to downstream tasks. Existing approaches merely leverage contrastive loss to learn instance-level discrimination. However, lack of category information will lead to hard-positive problem that constrains the generalization ability of this kind of methods. We find that the multi-task process of meta learning can provide a solution to this problem. In this paper, we propose a Meta-Contrastive Network (MCN), which combines the contrastive learning and meta learning, to enhance the learning ability of existing self-supervised approaches. Our method contains two training stages based on model-agnostic meta learning (MAML), each of which consists of a contrastive branch and a meta branch. Extensive evaluations demonstrate the effectiveness of our method. For two downstream tasks, i.e., video action recognition and video retrieval, MCN outperforms state-of-the-art approaches on UCF101 and HMDB51 datasets. To be more specific, with R(2+1)D backbone, MCN achieves Top-1 accuracies of 84.8% and 54.5% for video action recognition, as well as 52.5% and 23.7% for video retrieval.