Learning Generalizable Visual Representations via Self-Supervised Information Bottleneck

• Published in: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 5385 - 5389
• Main Authors: Liu, Xin, Li, Ya-li, Wang, Shengjin
• Format: Conference Proceeding
• Language: English
• Published: IEEE 14.04.2024
• Subjects: Acoustics; Brain modeling; Data models; Generalizable Representation Learning; Information Bottleneck; Instruments; Mutual Information; Representation learning; Self-Supervised Learning; Signal processing; Visual Representation Learning; Visualization
• ISSN: 2379-190X
• DOI: 10.1109/ICASSP48485.2024.10446545

Numerous approaches have recently emerged in the realm of self-supervised visual representation learning. While these methods have demonstrated empirical success, a theoretical foundation that understands and unifies these diverse techniques remains to be established. In this work, we draw inspiration from the principles underlying brain-based learning and propose a new method named self-supervised information bottleneck. Our method aims to maximize the mutual information between representations of views derived from the same image, while maintaining a minimal mutual information between the view and its corresponding representation at the same time. The brain-inspired method provides a unified information-theoretic perspective on various self-supervised approaches. This unified framework also empowers the model to learn generalizable visual representations for diverse downstream tasks and data distributions, achieving state-of-the-art performance across a wide variety of image and video tasks.