Active inference and agency: optimal control without cost functions

• Published in: Biological cybernetics Vol. 106; no. 8-9; pp. 523 - 541
• Main Authors: Friston, Karl, Samothrakis, Spyridon, Montague, Read
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer-Verlag 01.10.2012; Springer Nature B.V
• Subjects: Active control; Algorithms; Bayes Theorem; Bayesian analysis; Bioinformatics; Biological; Biomedical and Life Sciences; Biomedicine; Complex Systems; Computer Appl. in Life Sciences; Cost function; Cybernetics; Decision making; Decision Making - physiology; Filtering; Filtration; Inference; Markov analysis; Markov Chains; Mathematical models; Models, Neurological; Neurobiology; Neurosciences; Optimal control; Prospects; Action; Agency; Optimal control; Partially observable Markov decision processes; Bayesian; Inference; Free energy
• ISSN: 0340-1200; 1432-0770; 1432-0770
• DOI: 10.1007/s00422-012-0512-8

This paper describes a variational free-energy formulation of (partially observable) Markov decision problems in decision making under uncertainty. We show that optimal control can be cast as active inference . In active inference, both action and posterior beliefs about hidden states minimise a free energy bound on the negative log-likelihood of observed states, under a generative model. In this setting, reward or cost functions are absorbed into prior beliefs about state transitions and terminal states. Effectively, this converts optimal control into a pure inference problem, enabling the application of standard Bayesian filtering techniques. We then consider optimal trajectories that rest on posterior beliefs about hidden states in the future. Crucially, this entails modelling control as a hidden state that endows the generative model with a representation of agency. This leads to a distinction between models with and without inference on hidden control states; namely, agency-free and agency-based models, respectively.