Mutual Learning Knowledge Distillation Based on Multi-stage Multi-generative Adversarial Network

• Published in: Ji suan ji ke xue Vol. 49; no. 10; pp. 169 - 175
• Main Author: HUANG Zhong-hao, YANG Xing-yao, YU Jiong, GUO Liang, LI Xiang
• Format: Journal Article
• Language: Chinese
• Published: Editorial office of Computer Science 01.10.2022
• Subjects: mutual learning knowledge distillation|layer-wise greedy strategy|generative adversarial network|model compression|image classification
• ISSN: 1002-137X
• DOI: 10.11896/jsjkx.210800250

Aiming at the problems of insufficient knowledge distillation efficiency,single stage training methods,complex training processes and difficult convergence of traditional knowledge distillation methods in image classification tasks,this paper designs a mutual learning knowledge distillation based on multi-stage multi-generative adversarial networks(MS-MGANs).Firstly,the whole training process is divided into several stages,teacher models of different stages are obtained to guide student models to achieve better accuracy.Secondly,the layer-wise greedy strategy is introduced to replace the traditional end-to-end training mode,and the layer-wise training strategy based on convolution block is adopted to reduce the number of parameters to be optimized in each iteration process,and further improve the distillation efficiency of the model.Finally,a generative adversarial structure is introduced into the knowledge distillation framework,with the teacher model as the feature discriminator and the student model as the