Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision Transformers

• Published in: arXiv.org
• Main Authors: Ye, Peng, Huang, Yongqi, Tu, Chongjun, Li, Minglei, Chen, Tao, He, Tong, Ouyang, Wanli
• Format: Paper
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 25.12.2023
• Subjects: Datasets; Mathematical models; Parameters; Performance enhancement; Transformers

Fine-tuning pre-trained foundation models has gained significant popularity in various research fields. Existing methods for fine-tuning can be roughly divided into two categories, namely Parameter-Efficient Fine-Tuning and High-Performance Fine-Tuning. The former aims at improving efficiency, while the latter focuses on enhancing performance. Beyond these methods, we demonstrate that Partial Fine-Tuning can be an innovative and promising direction capable of concurrently enhancing both efficiency and accuracy. We first validate eight manually-defined partial fine-tuning strategies across kinds of datasets and vision transformer architectures, and find that some partial fine-tuning strategies (e.g., ffn only or attention only) can achieve better performance with fewer tuned parameters than full fine-tuning, and selecting appropriate layers is critical to partial fine-tuning. Thus, we propose a novel fine-tuned angle metric to guide the selection of appropriate layers for partial fine-tuning, making it flexible to be adapted to various scenarios for more practicable partial fine-tuning. Additionally, we show that partial fine-tuning can serve as a new dimension for Model Soups, improving both the model performance and generalization with fewer tuned parameters. Comprehensive experiments on a wide range of datasets and models validate the great potential of partial fine-tuning.