Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 24185 - 24198
• Main Authors: Chen, Zhe, Wu, Jiannan, Wang, Wenhai, Su, Weijie, Chen, Guo, Xing, Sen, Zhong, Muyan, Zhang, Qinglong, Zhu, Xizhou, Lu, Lewei, Li, Bin, Luo, Ping, Lu, Tong, Qiao, Yu, Dai, Jifeng
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Computer vision; Data models; Image recognition; Large language models; multi-modal; Question answering (information retrieval); Training; vision foundation model; vision-language model; Visualization
• ISSN: 1063-6919
• DOI: 10.1109/CVPR52733.2024.02283

The exponential growth of large language models (LLMs) has opened up numerous possibilities for multi-modal AGI systems. However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs. In this work, we design a large-scale vision-language foun-dation model (Intern VL), which scales up the vision foun-dation model to 6 billion parameters and progressively aligns it with the LLM, using web-scale image-text data from various sources. This model can be broadly applied to and achieve state-of-the-art performance on 32 generic visual-linguistic benchmarks including visual perception tasks such as image-level or pixel-level recognition, vision-language tasks such as zero-shot image/video classification, zero-shot image/video-text retrieval, and link with LLMs to create multi-modal dialogue systems. It has powerful visual capabilities and can be a good alternative to the ViT-22B. We hope that our research could contribute to the development of multi-modal large models.