End-to-end Autonomous Driving: Challenges and Frontiers

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. PP; pp. 1 - 20
• Main Authors: Chen, Li, Wu, Penghao, Chitta, Kashyap, Jaeger, Bernhard, Geiger, Andreas, Li, Hongyang
• Format: Journal Article
• Language: English
• Published: United States IEEE 30.07.2024
• Subjects: Autonomous Driving; Autonomous vehicles; Benchmark testing; End-to-end System Design; Imitation learning; Planning; Policy Learning; Simulation; Surveys; Task analysis; Trajectory
• ISSN: 0162-8828; 1939-3539; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2024.3435937

The autonomous driving community has witnessed a rapid growth in approaches that embrace an end-to-end algorithm framework, utilizing raw sensor input to generate vehicle motion plans, instead of concentrating on individual tasks such as detection and motion prediction. End-to-end systems, in comparison to modular pipelines, benefit from joint feature optimization for perception and planning. This field has flourished due to the availability of large-scale datasets, closed-loop evaluation, and the increasing need for autonomous driving algorithms to perform effectively in challenging scenarios. In this survey, we provide a comprehensive analysis of more than 270 papers, covering the motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. We delve into several critical challenges, including multi-modality, interpretability, causal confusion, robustness, and world models, amongst others. Additionally, we discuss current advancements in foundation models and visual pre-training, as well as how to incorporate these techniques within the end-to-end driving framework.We maintain an active repository that contains up-to-date literature and open-source projects at https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving .