Single frame 4D detection using deep fusion of camera image, imaging RADAR and LiDAR point cloud

• Main Authors: Tan, Ruoyu, Yalla, Veera Ganesh, Chen, Xu, Aithal, Ashwath, Rao, Ajaya H S, Deng, Huazeng
• Format: Patent
• Language: English
• Published: 07.09.2021
• Subjects: ANALOGOUS ARRANGEMENTS USING OTHER WAVES; CALCULATING; COMPUTING; CONTROLLING; COUNTING; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; IMAGE DATA PROCESSING OR GENERATION, IN GENERAL; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION ORRERADIATION OF RADIO WAVES; MEASURING; PHYSICS; RADIO DIRECTION-FINDING; RADIO NAVIGATION; REGULATING; SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES; TESTING

Embodiments of the present disclosure are directed to a method for object detection. The method includes receiving sensor data indicative of one or more objects for each of a camera subsystem, a LiDAR subsystem, and an imaging RADAR subsystem. The sensor data is received simultaneously and within one frame for each of the subsystems. The method also includes extracting one or more feature representations of the objects from camera image data, LiDAR point cloud data and imaging RADAR point cloud data and generating image feature maps, LiDAR feature maps and imaging RADAR feature maps. The method further includes combining the image feature maps, the LiDAR feature maps and the imaging RADAR feature maps to generate merged feature maps and generating object classification, object position, object dimensions, object heading and object velocity from the merged feature maps.