Dual discriminator adversarial distillation for data-free model compression

• Published in: International journal of machine learning and cybernetics Vol. 13; no. 5; pp. 1213 - 1230
• Main Authors: Zhao, Haoran, Sun, Xin, Dong, Junyu, Manic, Milos, Zhou, Huiyu, Yu, Hui
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.05.2022; Springer Nature B.V
• Subjects: Artificial Intelligence; Back propagation; Complex Systems; Computational Intelligence; Computer vision; Control; Datasets; Discriminators; Distillation; Engineering; Knowledge; Machine learning; Mechatronics; Neural networks; Original Article; Pattern Recognition; Robotics; Semantic segmentation; Semantics; Systems Biology; Teachers; Deep neural networks; Knowledge distillation; Model compression; Data-free; Image classification
• ISSN: 1868-8071; 1868-808X
• DOI: 10.1007/s13042-021-01443-0

Knowledge distillation has been widely used to produce portable and efficient neural networks which can be well applied on edge devices for computer vision tasks. However, almost all top-performing knowledge distillation methods need to access the original training data, which usually has a huge size and is often unavailable. To tackle this problem, we propose a novel data-free approach in this paper, named Dual Discriminator Adversarial Distillation (DDAD) to distill a neural network without the need of any training data or meta-data. To be specific, we use a generator to create samples through dual discriminator adversarial distillation, which mimics the original training data. The generator not only uses the pre-trained teacher’s intrinsic statistics in existing batch normalization layers but also obtains the maximum discrepancy from the student model. Then the generated samples are used to train the compact student network under the supervision of the teacher. The proposed method obtains an efficient student network which closely approximates its teacher network, without using the original training data. Extensive experiments are conducted to demonstrate the effectiveness of the proposed approach on CIFAR, Caltech101 and ImageNet datasets for classification tasks. Moreover, we extend our method to semantic segmentation tasks on several public datasets such as CamVid, NYUv2, Cityscapes and VOC 2012. To the best of our knowledge, this is the first work on generative model based data-free knowledge distillation on large-scale datasets such as ImageNet, Cityscapes and VOC 2012. Experiments show that our method outperforms all baselines for data-free knowledge distillation.