Revisiting the Transferability of Few-Shot Image Classification: A Frequency Spectrum Perspective

• Published in: Entropy (Basel, Switzerland) Vol. 26; no. 6; p. 473
• Main Authors: Zhang, Min, Wang, Zhitao, Wang, Donglin
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 29.05.2024; MDPI
• Subjects: causal perspective; Classification; Datasets; distribution shift; few-shot image classification; few-shot learning; Frequency spectrum; Image classification; Learning; Machine learning; Methods; Neural networks; Semantics; few-shot image classification; distribution shift; frequency spectrum; causal perspective; few-shot learning
• ISSN: 1099-4300; 1099-4300
• DOI: 10.3390/e26060473

Few-shot learning, especially few-shot image classification (FSIC), endeavors to recognize new categories using only a handful of labeled images by transferring knowledge from a model trained on base categories. Despite numerous efforts to address the challenge of deficient transferability caused by the distribution shift between the base and new classes, the fundamental principles remain a subject of debate. In this paper, we elucidate why a decline in performance occurs and what information is transferred during the testing phase, examining it from a frequency spectrum perspective. Specifically, we adopt causality on the frequency space for FSIC. With our causal assumption, non-causal frequencies (e.g., background knowledge) act as confounders between causal frequencies (e.g., object information) and predictions. Our experimental results reveal that different frequency components represent distinct semantics, and non-causal frequencies adversely affect transferability, resulting in suboptimal performance. Subsequently, we suggest a straightforward but potent approach, namely the Frequency Spectrum Mask (FRSM), to weight the frequency and mitigate the impact of non-causal frequencies. Extensive experiments demonstrate that the proposed FRSM method significantly enhanced the transferability of the FSIC model across nine testing datasets.