Atten-Adapter: A Unified Attention-Based Adapter for Efficient Tuning

Recently, more and more large pre-trained models have emerged. Several parameter-efficient tuning methods have been studied to transfer the prior knowledge of the pre-trained models to specific downstream tasks and achieve promising results. This paper proposes a simple yet effective method called A...

Full description

Saved in:
Bibliographic Details
Published in2023 IEEE International Conference on Image Processing (ICIP) pp. 1265 - 1269
Main Authors Li, Kaiwen, Gu, Wenzhe, Xue, Maixuan, Xiao, Jiahua, Shi, Dahu, Wei, Xing
Format Conference Proceeding
LanguageEnglish
Published IEEE 08.10.2023
Subjects
Adaptation models
Adapter
Attention
Image segmentation
Large pre-trained model
Parameter-efficient tuning
Prompt
Task analysis
Transformers
Tuning
Online AccessGet full text

Cover

More Information
Summary:Recently, more and more large pre-trained models have emerged. Several parameter-efficient tuning methods have been studied to transfer the prior knowledge of the pre-trained models to specific downstream tasks and achieve promising results. This paper proposes a simple yet effective method called Atten-Adapter. To the best of our knowledge, this is the first work that utilizes attention with learnable parameters as the internal structure of the adapter in the field of fine-tuning. The attention-based adapter can provide better information fusion ability and pay more attention to the global features compared to the MLP-based adapter. As a plug-and-play module, Atten-Adapter can be easily adapted to different types of vision models such as ConvNets and Transformer architectures in different tasks like classification and segmentation. Moreover, we demonstrate the generality of our proposed adapters by conducting experiments on language models. With small amounts of tunable parameters, our method achieves significant improvements compared to the previous state-of-the-art methods.
DOI:10.1109/ICIP49359.2023.10223170