Exploring replay

• Published in: Nature communications Vol. 16; no. 1; pp. 1657 - 20
• Main Authors: Antonov, Georgy, Dayan, Peter
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 15.02.2025; Nature Publishing Group; Nature Portfolio
• Subjects: 631/378/116/2396; 631/378/1788; 631/378/2649/1409; Agents (artificial intelligence); Algorithms; Animals; Choice Behavior - physiology; Decision making; Exploration; Exploratory behavior; Exploratory Behavior - physiology; Hippocampus; Hippocampus - physiology; Humanities and Social Sciences; Humans; Learning; Learning - physiology; Machine learning; Male; Modelling; Models, Neurological; multidisciplinary; Navigation behavior; Planning; Reinforcement, Psychology; Science; Science (multidisciplinary); Sleep and wakefulness; Spatial Navigation - physiology; Uncertainty; Wakefulness
• ISSN: 2041-1723; 2041-1723
• DOI: 10.1038/s41467-025-56731-y

Animals face uncertainty about their environments due to initial ignorance or subsequent changes. They therefore need to explore. However, the algorithmic structure of exploratory choices in the brain still remains largely elusive. Artificial agents face the same problem, and a venerable idea in reinforcement learning is that they can plan appropriate exploratory choices offline, during the equivalent of quiet wakefulness or sleep. Although offline processing in humans and other animals, in the form of hippocampal replay and preplay, has recently been the subject of highly informative modelling, existing methods only apply to known environments. Thus, they cannot predict exploratory replay choices during learning and/or behaviour in the face of uncertainty. Here, we extend an influential theory of hippocampal replay and examine its potential role in approximately optimal exploration, deriving testable predictions for the patterns of exploratory replay choices in a paradigmatic spatial navigation task. Our modelling provides a normative interpretation of the available experimental data suggestive of exploratory replay. Furthermore, we highlight the importance of sequence replay, and license a range of new experimental paradigms that should further our understanding of offline processing. Exploration is vital for efficient and successful learning. Here, the authors propose a normative account of the role of hippocampal replay in generating approximately optimal exploratory choices.