Saliency-guided meta-hallucinator for few-shot learning

• Published in: Science China. Information sciences Vol. 67; no. 10; p. 202103
• Main Authors: Zhang, Hongguang, Liu, Chun, Wang, Jiandong, Ma, Linru, Koniusz, Piotr, Torr, Philip H. S., Yang, Lin
• Format: Journal Article
• Language: English
• Published: Beijing Science China Press 01.10.2024; Springer Nature B.V
• Subjects: Anomalies; Computer Science; Deep learning; Hallucinations; Image classification; Image contrast; Image processors; Image quality; Information Systems and Communication Service; Machine learning; Research Paper; Salience; object recognition; anomaly detection; saliency detection; computer vision; few-shot learning
• ISSN: 1674-733X; 1869-1919
• DOI: 10.1007/s11432-023-4113-1

Learning novel object concepts from limited samples remains a considerable challenge in deep learning. The main directions for improving the few-shot learning models include (i) designing a stronger backbone, (ii) designing a powerful (dynamic) meta-classifier, and (iii) using a larger pre-training set obtained by generating or hallucinating additional samples from the small scale dataset. In this paper, we focus on item (iii) and present a novel meta-hallucination strategy. Presently, most image generators are based on a generative network (i.e., GAN) that generates new samples from the captured distribution of images. However, such networks require numerous annotated samples for training. In contrast, we propose a novel saliency-based end-to-end meta-hallucinator, where a saliency detector produces foregrounds and backgrounds of support images. Such images are fed into a two-stream network to hallucinate feature samples directly in the feature space by mixing foreground and background feature samples. Then, we propose several novel mixing strategies that improve the quality and diversity of hallucinated feature samples. Moreover, as not all saliency maps are meaningful or high quality, we further introduce a meta-hallucination controller that decides which foreground feature samples should participate in mixing with backgrounds. To our knowledge, we are the first to leverage saliency detection for few-shot learning. Our proposed network achieves state-of-the-art results on publicly available few-shot image classification and anomaly detection benchmarks, and outperforms competing sample mixing strategies such as the so-called Manifold Mixup.