Few-shot incremental learning with continual prototype calibration for remote sensing image fine-grained classification

• Published in: ISPRS journal of photogrammetry and remote sensing Vol. 196; pp. 210 - 227
• Main Authors: Zhu, Zining, Wang, Peijin, Diao, Wenhui, Yang, Jinze, Wang, Hongqi, Sun, Xian
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.02.2023
• Subjects: Continual prototype calibration; Few-shot learning; Fine-grained classification; Incremental learning; Remote sensing image; Remote sensing image; Incremental learning; Few-shot learning; Continual prototype calibration; Fine-grained classification
• ISSN: 0924-2716; 1872-8235
• DOI: 10.1016/j.isprsjprs.2022.12.024

With the rapid acquisition of remote sensing (RS) data, new categories of objects continue to emerge, and some categories can only obtain a few training samples. Thus, few-shot class-incremental learning (FSCIL) has received intense attention in recent years. Nevertheless, the existing FSCIL methods are difficult to adapt to RS characteristics with high inter-class similarity, large intra-class differences, and complex backgrounds. In this paper, we propose the FSCIL method called continual prototype calibration (CPC), which applies the decoupled learning strategy to cope with the demand to retrain the model when new few-shot categories appear. It can also address the tedious problems in fine-grained RS images by calibrating inter-class and intra-class prototypes. Concretely, considering the severe issue of inter-class confusion, we introduce a prototype separability module (PSM) to update the distribution of inter-class prototypes. In this way, we can distinguish different categories more clearly by weakening their similarities, and the phenomenon of catastrophic forgetting can also be effectively mitigated. Furthermore, to generate more representative and accurate intra-class prototypes, we design a Meta-Network based on foreground enhancement mechanism (FEM). The Meta-Network can boost the learning ability of the model for few-shot data and suppress the occurrence of overfitting by training on task-based data. And the FEM enhances the object features by filtering out redundant background information, facilitating fine-grained classification. Our CPC method is the first FSCIL method applied to optical RS images, and we verify its effectiveness on three datasets. Extensive experiments illustrate that our method achieves state-of-the-art performance in FSCIL of fine-grained RS images. The code is available at https://github.com/ningerhhh/CPC.