itKD: Interchange Transfer-based Knowledge Distillation for 3D Object Detection

• Published in: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 13540 - 13549
• Main Authors: Cho, Hyeon, Choi, Junyong, Baek, Geonwoo, Hwang, Wonjun
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2023
• Subjects: Computer vision; Detectors; Efficient and scalable vision; Knowledge engineering; Object detection; Point cloud compression; Solid modeling; Three-dimensional displays
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52729.2023.01301

Point-cloud based 3D object detectors recently have achieved remarkable progress. However, most studies are limited to the development of network architectures for improving only their accuracy without consideration of the computational efficiency. In this paper, we first propose an autoencoder-style framework comprising channel-wise compression and decompression via interchange transfer-based knowledge distillation. To learn the map-view feature of a teacher network, the features from teacher and student networks are independently passed through the shared autoencoder; here, we use a compressed representation loss that binds the channel-wised compression knowledge from both student and teacher networks as a kind of regularization. The decompressed features are transferred in opposite directions to reduce the gap in the interchange reconstructions. Lastly, we present an head attention loss to match the 3D object detection information drawn by the multi-head self-attention mechanism. Through extensive experiments, we verify that our method can train the lightweight model that is well-aligned with the 3D point cloud detection task and we demonstrate its superiority using the well-known public datasets; e.g., Waymo and nuScenes. 1 1 Our code is available at https://github.com/hyeon-jo/interchange-transfer-KD.