V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving

• Published in: IEEE robotics and automation letters Vol. 7; no. 4; pp. 10914 - 10921
• Main Authors: Li, Yiming, Ma, Dekun, An, Ziyan, Wang, Zixun, Zhong, Yiqi, Chen, Siheng, Feng, Chen
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.10.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Autonomous vehicles; Benchmarks; Cameras; Collaboration; data sets for robotic vision; Datasets; Deep learning for visual perception; Laser radar; multi-robot systems; Multiagent systems; Perception; Robot sensing systems; Segmentation; Semantics; Task analysis
• ISSN: 2377-3766; 2377-3766
• DOI: 10.1109/LRA.2022.3192802

Vehicle-to-everything (V2X) communication techniques enable the collaboration between vehicles and many other entities in the neighboring environment, which could fundamentally improve the perception system for autonomous driving. However, the lack of a public dataset significantly restricts the research progress of collaborative perception. To fill this gap, we present V2X-Sim, a comprehensive simulated multi-agent perception dataset for V2X-aided autonomous driving. V2X-Sim provides: (1) multi-agent sensor recordings from the road-side unit (RSU) and multiple vehicles that enable collaborative perception, (2) multi-modality sensor streams that facilitate multi-modality perception, and (3) diverse ground truths that support various perception tasks. Meanwhile, we build an open-source testbed and provide a benchmark for the state-of-the-art collaborative perception algorithms on three tasks, including detection, tracking and segmentation. V2X-Sim seeks to stimulate collaborative perception research for autonomous driving before realistic datasets become widely available.