InfraDet3D: Multi-Modal 3D Object Detection based on Roadside Infrastructure Camera and LiDAR Sensors
Current multi-modal object detection approaches focus on the vehicle domain and are limited in the perception range and the processing capabilities. Roadside sensor units (RSUs) introduce a new domain for perception systems and leverage altitude to observe traffic. Cameras and LiDARs mounted on gant...
Saved in:
Published in | 2023 IEEE Intelligent Vehicles Symposium (IV) pp. 1 - 8 |
---|---|
Main Authors | , , , , , , |
Format | Conference Proceeding |
Language | English |
Published |
IEEE
04.06.2023
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Current multi-modal object detection approaches focus on the vehicle domain and are limited in the perception range and the processing capabilities. Roadside sensor units (RSUs) introduce a new domain for perception systems and leverage altitude to observe traffic. Cameras and LiDARs mounted on gantry bridges increase the perception range and produce a full digital twin of the traffic. In this work, we introduce InfraDet3D, a multi-modal 3D object detector for roadside infrastructure sensors. We fuse two LiDARs using early fusion and further incorporate detections from monocular cameras to increase the robustness and to detect small objects. Our monocular 3D detection module uses HD maps to ground object yaw hypotheses, improving the final perception results. The perception framework is deployed on a real-world intersection that is part of the A9 Test Stretch in Munich, Germany. We perform several ablation studies and experiments and show that fusing two LiDARs with two cameras leads to an improvement of +1.90 mAP compared to a camera-only solution. We evaluate our results on the A9 infrastructure dataset and achieve 68.48 mAP on the test set. The dataset and code will be available at https://a9-dataset.com to allow the research community to further improve the perception results and make autonomous driving safer. |
---|---|
AbstractList | Current multi-modal object detection approaches focus on the vehicle domain and are limited in the perception range and the processing capabilities. Roadside sensor units (RSUs) introduce a new domain for perception systems and leverage altitude to observe traffic. Cameras and LiDARs mounted on gantry bridges increase the perception range and produce a full digital twin of the traffic. In this work, we introduce InfraDet3D, a multi-modal 3D object detector for roadside infrastructure sensors. We fuse two LiDARs using early fusion and further incorporate detections from monocular cameras to increase the robustness and to detect small objects. Our monocular 3D detection module uses HD maps to ground object yaw hypotheses, improving the final perception results. The perception framework is deployed on a real-world intersection that is part of the A9 Test Stretch in Munich, Germany. We perform several ablation studies and experiments and show that fusing two LiDARs with two cameras leads to an improvement of +1.90 mAP compared to a camera-only solution. We evaluate our results on the A9 infrastructure dataset and achieve 68.48 mAP on the test set. The dataset and code will be available at https://a9-dataset.com to allow the research community to further improve the perception results and make autonomous driving safer. |
Author | Wang, Bohan Birkner, Joseph Zimmer, Walter Petrovski, Stefan Brucker, Marcel Tung Nguyen, Huu Knoll, Alois C. |
Author_xml | – sequence: 1 givenname: Walter surname: Zimmer fullname: Zimmer, Walter email: walter.zimmer@tum.de organization: Technical University of Munich, TUM,School of Computation, Information and Technology (CIT),Department of Informatics,Garching-Hochbrueck,Germany,85748 – sequence: 2 givenname: Joseph surname: Birkner fullname: Birkner, Joseph organization: Technical University of Munich, TUM,School of Computation, Information and Technology (CIT),Department of Informatics,Garching-Hochbrueck,Germany,85748 – sequence: 3 givenname: Marcel surname: Brucker fullname: Brucker, Marcel organization: Technical University of Munich, TUM,School of Computation, Information and Technology (CIT),Department of Informatics,Garching-Hochbrueck,Germany,85748 – sequence: 4 givenname: Huu surname: Tung Nguyen fullname: Tung Nguyen, Huu organization: Technical University of Munich, TUM,School of Computation, Information and Technology (CIT),Department of Informatics,Garching-Hochbrueck,Germany,85748 – sequence: 5 givenname: Stefan surname: Petrovski fullname: Petrovski, Stefan organization: Technical University of Munich, TUM,School of Computation, Information and Technology (CIT),Department of Informatics,Garching-Hochbrueck,Germany,85748 – sequence: 6 givenname: Bohan surname: Wang fullname: Wang, Bohan organization: Technical University of Munich, TUM,School of Computation, Information and Technology (CIT),Department of Informatics,Garching-Hochbrueck,Germany,85748 – sequence: 7 givenname: Alois C. surname: Knoll fullname: Knoll, Alois C. organization: Technical University of Munich, TUM,School of Computation, Information and Technology (CIT),Department of Informatics,Garching-Hochbrueck,Germany,85748 |
BookMark | eNo1kMtqwzAURNXSQpM0f9CCfsCp7pVlS92FOG0DCYH0sQ2yfAUKiV0kZ9G_r-ljdQYOM4sZs6u2a4mxexAzAGEeVh9KgcIZCpQzEKCLEuUFm5rSaKmEzAsDxSUbYZFjViLkN2yc0kEIpRBhxGjV-mgr6mX1yDfnYx-yTdfYI5cV39YHcj0f5IDQtby2iRo-hF1nmxQa4j_t1Mez68-R-MKeKFpu24avQzXf8VdqUxfTLbv29pho-scJe39avi1esvX2ebWYr7OAIu8zqRFr8CiVk06JBnQO4MAal9egrTXkvJauFLm2oKhEU4LyovBSDdYXcsLufncDEe0_YzjZ-LX_v0V-Aw5nV2M |
ContentType | Conference Proceeding |
DBID | 6IE 6IL CBEJK RIE RIL |
DOI | 10.1109/IV55152.2023.10186723 |
DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present |
DatabaseTitleList | |
Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ sourceTypes: Publisher |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Engineering |
EISBN | 9798350346916 |
EISSN | 2642-7214 |
EndPage | 8 |
ExternalDocumentID | 10186723 |
Genre | orig-research |
GrantInformation_xml | – fundername: Ministry of Education funderid: 10.13039/100009950 |
GroupedDBID | 6IE 6IF 6IH 6IK 6IL 6IN AAJGR ACGFS ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IPLJI M43 OCL RIE RIL RNS |
ID | FETCH-LOGICAL-i204t-3822b1f235c3c50d18411c1a9c4b18aa9ecf83c7048a15e729715f06f358aaf63 |
IEDL.DBID | RIE |
IngestDate | Wed Jun 26 19:25:17 EDT 2024 |
IsPeerReviewed | false |
IsScholarly | true |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-i204t-3822b1f235c3c50d18411c1a9c4b18aa9ecf83c7048a15e729715f06f358aaf63 |
PageCount | 8 |
ParticipantIDs | ieee_primary_10186723 |
PublicationCentury | 2000 |
PublicationDate | 2023-June-4 |
PublicationDateYYYYMMDD | 2023-06-04 |
PublicationDate_xml | – month: 06 year: 2023 text: 2023-June-4 day: 04 |
PublicationDecade | 2020 |
PublicationTitle | 2023 IEEE Intelligent Vehicles Symposium (IV) |
PublicationTitleAbbrev | IV |
PublicationYear | 2023 |
Publisher | IEEE |
Publisher_xml | – name: IEEE |
SSID | ssj0055221 |
Score | 2.336886 |
Snippet | Current multi-modal object detection approaches focus on the vehicle domain and are limited in the perception range and the processing capabilities. Roadside... |
SourceID | ieee |
SourceType | Publisher |
StartPage | 1 |
SubjectTerms | 3D Perception Autonomous Driving Bridges Camera-LiDAR Fusion Detectors Infrastructure Sensors Laser radar Object detection Point cloud compression Roadside Sensors Snow Three-dimensional displays |
Title | InfraDet3D: Multi-Modal 3D Object Detection based on Roadside Infrastructure Camera and LiDAR Sensors |
URI | https://ieeexplore.ieee.org/document/10186723 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3JTsMwELVoT3BhK2KXD1yTeomzcEMNVYtoQYWi3iqvUgVKUEkvfD2207BJSNwsW5aTGSWzeN4bAC4YEpnhWAaxNnEQJQkKRJKZIOMqkWnKTUodGnk0jgfT6GbGZmuwusfCaK198ZkO3dDf5atSrlyqrOvYpeKE0BZopYjUYK3mt8usI4HXEB2Msu7wyfoCzEGtCA2bjT9aqHgL0t8G4-bsunDkOVxVIpTvv2gZ__1wO6DzBdaD959maBds6GIPbH3jGdwHeliYJc91RfNL6CG3wahU_AXSHN4Jl4mBdtEXZRXQ2TUF7WBScuWaeUK_uyaaXS017HGXyIK8UPB2kV9N4IMNhcvlWwdM-9ePvUGw7q8QLAiKqoBa50BgQyiTVDKkbLCHscQ8k5HAKeeZllZVMrEfOcdMWzc8wcyg2FBmV01MD0C7KAt9CCCVcUakVbCO0ihSRGjEY8YTG7AQQbk8Ah0nsflrTaExb4R1_Mf8Cdh0ivM1WdEpaNtX1GfW-lfi3Gv9AyJXriQ |
link.rule.ids | 310,311,786,790,795,796,802,27958,55109 |
linkProvider | IEEE |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3JTsMwELWgHIALWxE7PnBNGsdxFm6ooWqhLai0iFtlO7ZUgRJU0gtfz9hp2CQkblYmVmJPklky7w1CF8wTieZEOqHSoRNEkeeIKNFOwrNIxjHXMTVo5MEw7E6Cmyf2tASrWyyMUsoWnynXDO2__KyQC5Mqaxl2qTDy6SpaA0PvJRVcq_7wMnAlyBKkA8JW7xFOYgZs5VO3nvqjiYq1IZ0tNKyvXpWOPLuLUrjy_Rcx479vbxs1v-B6-P7TEO2gFZXvos1vTIN7SPVyPeepKml6iS3o1hkUGX_BNMV3wuRiMAhtWVaOjWXLMAxGBc9MO09sZ1dUs4u5wm1uUlmY5xnuz9KrEX6AYLiYvzXRpHM9bnedZYcFZ-Z7QelQcA8E0T5lkkrmZRDuESIJT2QgSMx5oiQoS0bwmnPCFDjiEWHaCzVlINUh3UeNvMjVAcJUhokvQcUqiIMg84XyeMh4BCGLLyiXh6hpdmz6WpFoTOvNOvrj-Dla744H_Wm_N7w9RhtGibZCKzhBDViuOgVfoBRn9gn4AIPHsXo |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2023+IEEE+Intelligent+Vehicles+Symposium+%28IV%29&rft.atitle=InfraDet3D%3A+Multi-Modal+3D+Object+Detection+based+on+Roadside+Infrastructure+Camera+and+LiDAR+Sensors&rft.au=Zimmer%2C+Walter&rft.au=Birkner%2C+Joseph&rft.au=Brucker%2C+Marcel&rft.au=Tung+Nguyen%2C+Huu&rft.date=2023-06-04&rft.pub=IEEE&rft.eissn=2642-7214&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1109%2FIV55152.2023.10186723&rft.externalDocID=10186723 |