Tip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification

• Published in: Computer Vision - ECCV 2022 Vol. 13695; pp. 493 - 510
• Main Authors: Zhang, Renrui, Zhang, Wei, Fang, Rongyao, Gao, Peng, Li, Kunchang, Dai, Jifeng, Qiao, Yu, Li, Hongsheng
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer 2022; Springer Nature Switzerland
• Series: Lecture Notes in Computer Science
• Subjects: Cache model; Few-shot classification; Vision-language learning
• ISBN: 9783031198328; 3031198328
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-031-19833-5_29

Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations using large-scale image-text pairs. It shows impressive performance on downstream tasks by zero-shot knowledge transfer. To further enhance CLIP’s adaption capability, existing methods proposed to fine-tune additional learnable modules, which significantly improves the few-shot performance but introduces extra training time and computational resources. In this paper, we propose a Training-free adaption method for CLIP to conduct few-shot classification, termed as Tip-Adapter, which not only inherits the training-free advantage of zero-shot CLIP but also performs comparably to those training-required approaches. Tip-Adapter constructs the adapter via a key-value cache model from the few-shot training set, and updates the prior knowledge encoded in CLIP by feature retrieval. On top of that, the performance of Tip-Adapter can be further boosted to be state-of-the-art on ImageNet by fine-tuning the cache model for 10× $$\times $$ fewer epochs than existing methods, which is both effective and efficient. We conduct extensive experiments of few-shot classification on 11 datasets to demonstrate the superiority of our proposed methods. Code is released at https://github.com/gaopengcuhk/Tip-Adapter.