Radar and Lidar Deep Fusion: Providing Doppler Contexts to Time-of-Flight Lidar

• Published in: IEEE sensors journal Vol. 23; no. 20; pp. 25587 - 25600
• Main Authors: Jin, Yi, Kuang, Yongshao, Hoffmann, Marcel, Schusler, Christian, Deligiannis, Anastasios, Fuentes-Michel, Juan-Carlos, Vossiek, Martin
• Format: Journal Article
• Language: English
• Published: New York IEEE 15.10.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Classification; Convolutional neural network (CNN); deep learning; Doppler effect; Doppler radar; Laser radar; Lidar; Motion perception; Multisensor fusion; object detection; Object motion; Object recognition; Pedestrians; Proposals; Radar; Radar detection; Radar imaging; Roads; sensor fusion; Sensors
• ISSN: 1530-437X; 1558-1748
• DOI: 10.1109/JSEN.2023.3313093

This work proposes a novel sensor fusion-based, single-frame, multiclass object detection method for road users, including vehicles, pedestrians, and cyclists, in which a deep fusion occurs between the lidar point cloud (PC) and the corresponding Doppler contexts, namely, the Doppler features, from the radar cube. Based on convolutional neural networks (CNNs), the method consists of two stages: In the first stage, region proposals are generated from the voxelized lidar PC, and relying on these proposals, Doppler contexts are cropped from the radar cube. In the second stage, using fused features from the lidar and radar, the method achieves object detection and object motion status classification tasks. When evaluated with measurements in inclement conditions, which are generated by a foggification model from real-life measurements, in terms of the intersection over union (IoU) metric, the proposed method outperforms the lidar-based network by a large margin for vulnerable road users, namely, 4.5% and 6.1% improvement for pedestrians and cyclists, respectively. In addition, it achieves 87% <inline-formula> <tex-math notation="LaTeX">{F}_{{1}} </tex-math></inline-formula> score (81.6% precision and 93.1% recall) for single-frame, object motion status classification.