Outracing champion Gran Turismo drivers with deep reinforcement learning

• Published in: Nature (London) Vol. 602; no. 7896; pp. 223 - 228
• Main Authors: Wurman, Peter R., Barrett, Samuel, Kawamoto, Kenta, MacGlashan, James, Subramanian, Kaushik, Walsh, Thomas J., Capobianco, Roberto, Devlic, Alisa, Eckert, Franziska, Fuchs, Florian, Gilpin, Leilani, Khandelwal, Piyush, Kompella, Varun, Lin, HaoChih, MacAlpine, Patrick, Oller, Declan, Seno, Takuma, Sherstan, Craig, Thomure, Michael D., Aghabozorgi, Houmehr, Barrett, Leon, Douglas, Rory, Whitehead, Dion, Dürr, Peter, Stone, Peter, Spranger, Michael, Kitano, Hiroaki
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 10.02.2022; Nature Publishing Group
• Subjects: 639/705/1041; 639/705/117; Agents (artificial intelligence); Algorithms; Artificial intelligence; Automobile Driving - standards; Automobile racing; Competitive Behavior; Computer & video games; Deep Learning; Dynamical systems; Expected values; Human performance; Humanities and Social Sciences; Humans; Integrated control; Learning algorithms; Machine learning; Multiagent systems; multidisciplinary; Neural networks; Nonlinear control; Norms; Race cars; Reagents; Reinforcement; Reinforcement, Psychology; Reward; Robotics; Science; Science (multidisciplinary); Sports - standards; Tactics; Vehicles; Velocity; Video Games
• ISSN: 0028-0836; 1476-4687
• DOI: 10.1038/s41586-021-04357-7

Many potential applications of artificial intelligence involve making real-time decisions in physical systems while interacting with humans. Automobile racing represents an extreme example of these conditions; drivers must execute complex tactical manoeuvres to pass or block opponents while operating their vehicles at their traction limits 1 . Racing simulations, such as the PlayStation game Gran Turismo, faithfully reproduce the non-linear control challenges of real race cars while also encapsulating the complex multi-agent interactions. Here we describe how we trained agents for Gran Turismo that can compete with the world’s best e-sports drivers. We combine state-of-the-art, model-free, deep reinforcement learning algorithms with mixed-scenario training to learn an integrated control policy that combines exceptional speed with impressive tactics. In addition, we construct a reward function that enables the agent to be competitive while adhering to racing’s important, but under-specified, sportsmanship rules. We demonstrate the capabilities of our agent, Gran Turismo Sophy, by winning a head-to-head competition against four of the world’s best Gran Turismo drivers. By describing how we trained championship-level racers, we demonstrate the possibilities and challenges of using these techniques to control complex dynamical systems in domains where agents must respect imprecisely defined human norms. Using the game Gran Turismo, an agent was trained with a combination of deep reinforcement learning algorithms and specialized training scenarios, demonstrating success against championship-level human racers.