Deep Mutual Learning

Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, in order to meet the low-memory or fast execution requirements. In this paper,...

Full description

Saved in:
Bibliographic Details
Published in2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 4320 - 4328
Main Authors Zhang, Ying, Xiang, Tao, Hospedales, Timothy M., Lu, Huchuan
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.06.2018
Subjects
Computational modeling
Entropy
Neural networks
Optimization
Supervised learning
Task analysis
Training
Online AccessGet full text

Cover

More Information
Summary:Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, in order to meet the low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy. Different from the one-way transfer between a static pre-defined teacher and a student in model distillation, with DML, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on both category and instance recognition tasks. Surprisingly, it is revealed that no prior powerful teacher network is necessary - mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher.
ISSN:1063-6919
DOI:10.1109/CVPR.2018.00454