PointPainting: 3D Object Detection Aided by Semantic Image Information

• Published in: Sensors (Basel, Switzerland) Vol. 23; no. 5; p. 2868
• Main Authors: Gao, Zhentong, Wang, Qiantong, Pan, Zongxu, Zhai, Zhenyu, Long, Hui
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 06.03.2023; MDPI
• Subjects: 3D object detection; Alliances; data fusion; deep learning; Image processing; Methods; multi modal; Object recognition (Computers); Optical radar; Pattern recognition; semantic segmentation; Semantics; Sensors; China; data fusion; deep learning; multi modal; semantic segmentation; 3D object detection
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s23052868

A multi-modal 3D object-detection method, based on data from cameras and LiDAR, has become a subject of research interest. PointPainting proposes a method for improving point-cloud-based 3D object detectors using semantic information from RGB images. However, this method still needs to improve on the following two complications: first, there are faulty parts in the image semantic segmentation results, leading to false detections. Second, the commonly used anchor assigner only considers the intersection over union (IoU) between the anchors and ground truth boxes, meaning that some anchors contain few target LiDAR points assigned as positive anchors. In this paper, three improvements are suggested to address these complications. Specifically, a novel weighting strategy is proposed for each anchor in the classification loss. This enables the detector to pay more attention to anchors containing inaccurate semantic information. Then, SegIoU, which incorporates semantic information, instead of IoU, is proposed for the anchor assignment. SegIoU measures the similarity of the semantic information between each anchor and ground truth box, avoiding the defective anchor assignments mentioned above. In addition, a dual-attention module is introduced to enhance the voxelized point cloud. The experiments demonstrate that the proposed modules obtained significant improvements in various methods, consisting of single-stage PointPillars, two-stage SECOND-IoU, anchor-base SECOND, and an anchor-free CenterPoint on the KITTI dataset.