Student Network Learning via Evolutionary Knowledge Distillation

• Published in: IEEE transactions on circuits and systems for video technology Vol. 32; no. 4; pp. 2251 - 2263
• Main Authors: Zhang, Kangkai, Zhang, Chunhui, Li, Shikun, Zeng, Dan, Ge, Shiming
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.04.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Data mining; Data models; deep learning; Distillation; Evolution; Germanium; Knowledge; Knowledge distillation; Knowledge management; Knowledge representation; Knowledge transfer; Learning; Predictive models; Teachers; teacher–student learning; Training

Knowledge distillation provides an effective way to transfer knowledge via teacher-student learning, where most existing distillation approaches apply a fixed pre-trained model as teacher to supervise the learning of student network. This manner usually brings in a big capability gap between teacher...