Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models

• Published in: IEEE transactions on circuits and systems for video technology Vol. 33; no. 9; p. 1
• Main Authors: Ma, Chengcheng, Liu, Yang, Deng, Jiankang, Xie, Lingxi, Dong, Weiming, Xu, Changsheng
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.09.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Back propagation; Data models; Eigenvectors; Gradient flow; gradient projection; Image segmentation; Object recognition; Optimization; overfitting; prompt tuning; Semantic segmentation; subspace learning; Task analysis; Training; Training data; Tuning; Vision-language model; Visualization

Pretrained vision-language models (VLMs) such as CLIP have shown impressive generalization capability in downstream vision tasks with appropriate text prompts. Instead of designing prompts manually, Context Optimization (CoOp) has been recently proposed to learn continuous prompts using task-specifi...