Identification of animal behavioral strategies by inverse reinforcement learning

• Published in: PLoS computational biology Vol. 14; no. 5; p. e1006122
• Main Authors: Yamaguchi, Shoichiro, Naoki, Honda, Ikeda, Muneki, Tsukada, Yuki, Nakano, Shunji, Mori, Ikue, Ishii, Shin
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 02.05.2018; Public Library of Science (PLoS)
• Subjects: Animal behavior; Animals; Behavior; Behavior, Animal - physiology; Biology; Biology and Life Sciences; Caenorhabditis elegans; Caenorhabditis elegans - physiology; Computational Biology; Cultivation; Data processing; Decision making; Decision Making - physiology; Dopamine; Ecology and Environmental Sciences; Food; Funding; Informatics; Information processing; Inverse problems; Laboratories; Learning - physiology; Mathematical models; Migration; Nematodes; Nervous system; Neural networks; Neurobiology; Neurosciences; Physical Sciences; Reinforcement; Reinforcement, Psychology; Research and Analysis Methods; Social Sciences; Taxis Response - physiology; Temperature; Temperature effects; Thermotaxis; Time series; Japan
• ISSN: 1553-7358; 1553-734X; 1553-7358
• DOI: 10.1371/journal.pcbi.1006122

Animals are able to reach a desired state in an environment by controlling various behavioral patterns. Identification of the behavioral strategy used for this control is important for understanding animals' decision-making and is fundamental to dissect information processing done by the nervous system. However, methods for quantifying such behavioral strategies have not been fully established. In this study, we developed an inverse reinforcement-learning (IRL) framework to identify an animal's behavioral strategy from behavioral time-series data. We applied this framework to C. elegans thermotactic behavior; after cultivation at a constant temperature with or without food, fed worms prefer, while starved worms avoid the cultivation temperature on a thermal gradient. Our IRL approach revealed that the fed worms used both the absolute temperature and its temporal derivative and that their behavior involved two strategies: directed migration (DM) and isothermal migration (IM). With DM, worms efficiently reached specific temperatures, which explains their thermotactic behavior when fed. With IM, worms moved along a constant temperature, which reflects isothermal tracking, well-observed in previous studies. In contrast to fed animals, starved worms escaped the cultivation temperature using only the absolute, but not the temporal derivative of temperature. We also investigated the neural basis underlying these strategies, by applying our method to thermosensory neuron-deficient worms. Thus, our IRL-based approach is useful in identifying animal strategies from behavioral time-series data and could be applied to a wide range of behavioral studies, including decision-making, in other organisms.