Multi-view deep neural network for LiDAR perception

• Main Authors: Wehr, David, Pehserl, Joachim, Wekel, Tilman, Oldja, Ryan, Eden, Ibrahim, Nister, David, Bhargava, Ruchi, Smolyanskiy, Nikolai, Chen, Ke, Popov, Alexander
• Format: Patent
• Language: English
• Published: 03.09.2024
• Subjects: ANALOGOUS ARRANGEMENTS USING OTHER WAVES; CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE ORDIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; CONTROLLING; COUNTING; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; IMAGE DATA PROCESSING OR GENERATION, IN GENERAL; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION ORRERADIATION OF RADIO WAVES; MEASURING; PERFORMING OPERATIONS; PHYSICS; RADIO DIRECTION-FINDING; RADIO NAVIGATION; REGULATING; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TOTHE CONTROL OF A PARTICULAR SUB-UNIT; SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES; TESTING; TRANSPORTING; VEHICLES IN GENERAL

A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.