Autonomous navigation of stratospheric balloons using reinforcement learning

• Published in: Nature (London) Vol. 588; no. 7836; pp. 77 - 82
• Main Authors: Bellemare, Marc G., Candido, Salvatore, Castro, Pablo Samuel, Gong, Jun, Machado, Marlos C., Moitra, Subhodeep, Ponda, Sameera S., Wang, Ziyu
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 03.12.2020; Nature Publishing Group
• Subjects: 639/166/984; 639/705/117; Algorithms; Altitude; Aquatic birds; Autonomous navigation; Balloons; Controllers; Expected values; Flight control systems; Forecast errors; Humanities and Social Sciences; Intelligent agents; Learning; Meteorological balloons; multidisciplinary; Reinforcement; Science; Science (multidisciplinary); Stratospheric balloons; Stratospheric winds; Superpressure balloons; Wind; Wind measurement; Wind speed
• ISSN: 0028-0836; 1476-4687
• DOI: 10.1038/s41586-020-2939-8

Efficiently navigating a superpressure balloon in the stratosphere 1 requires the integration of a multitude of cues, such as wind speed and solar elevation, and the process is complicated by forecast errors and sparse wind measurements. Coupled with the need to make decisions in real time, these factors rule out the use of conventional control techniques 2 , 3 . Here we describe the use of reinforcement learning 4 , 5 to create a high-performing flight controller. Our algorithm uses data augmentation 6 , 7 and a self-correcting design to overcome the key technical challenge of reinforcement learning from imperfect data, which has proved to be a major obstacle to its application to physical systems 8 . We deployed our controller to station Loon superpressure balloons at multiple locations across the globe, including a 39-day controlled experiment over the Pacific Ocean. Analyses show that the controller outperforms Loon’s previous algorithm and is robust to the natural diversity in stratospheric winds. These results demonstrate that reinforcement learning is an effective solution to real-world autonomous control problems in which neither conventional methods nor human intervention suffice, offering clues about what may be needed to create artificially intelligent agents that continuously interact with real, dynamic environments. Data augmentation and a self-correcting design are used to develop a reinforcement-learning algorithm for the autonomous navigation of Loon superpressure balloons in challenging stratospheric weather conditions.