Attention-based Knowledge Distillation in Scene Recognition: the Impact of a DCT-driven Loss

Knowledge Distillation (KD) is a strategy for the definition of a set of transferability gangways to improve the efficiency of Convolutional Neural Networks. Feature-based Knowledge Distillation is a subfield of KD that relies on intermediate network representations, either unaltered or depth-reduce...

Full description

Saved in:
Bibliographic Details
Published inIEEE transactions on circuits and systems for video technology Vol. 33; no. 9; p. 1
Main Authors Lopez-Cifuentes, Alejandro, Escudero-Vinolo, Marcos, Bescos, Jesus, SanMiguel, Juan C.
Format Journal Article
LanguageEnglish
Published New York IEEE 01.09.2023
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects
2D frequency transform
Artificial neural networks
Computational modeling
convolutional neural networks
deep learning
Discrete cosine transforms
Distillation
Feature extraction
Knowledge distillation
Knowledge engineering
Multi-attention
Recognition
Scene recognition
Semantics
Task analysis
Training
Two dimensional analysis
Online AccessGet full text

Cover

More Information
Summary:Knowledge Distillation (KD) is a strategy for the definition of a set of transferability gangways to improve the efficiency of Convolutional Neural Networks. Feature-based Knowledge Distillation is a subfield of KD that relies on intermediate network representations, either unaltered or depth-reduced via maximum activation maps, as the source knowledge. In this paper, we propose and analyze the use of a 2D frequency transform of the activation maps before transferring them. We pose that-by using global image cues rather than pixel estimates, this strategy enhances knowledge transferability in tasks such as scene recognition, defined by strong spatial and contextual relationships between multiple and varied concepts. To validate the proposed method, an extensive evaluation of the state of the art in scene recognition is presented. Experimental results provide strong evidence that the proposed strategy enables the student network to better focus on the relevant image areas learnt by the teacher network, hence leading to better descriptive features and higher transferred performance than every other state-of-the-art alternative. We publicly release the training and evaluation framework used in this paper at http://www-vpu.eps. uam.es/publications/DCTBasedKDForSceneRecognition.
ISSN:1051-8215
1558-2205
DOI:10.1109/TCSVT.2023.3250031