Controlling blood glucose variability under uncertainty using reinforcement learning and Gaussian processes

• Published in: Applied soft computing Vol. 35; pp. 310 - 332
• Main Authors: De Paula, Mariano, Ávila, Luis Omar, Martínez, Ernesto C.
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.10.2015
• Subjects: Artificial pancreas; Diabetes; Gaussian processes; Policy iteration; Reinforcement learning; Stochastic optimal control; Artificial pancreas; Policy iteration; Diabetes; Stochastic optimal control; Gaussian processes; Reinforcement learning
• ISSN: 1568-4946; 1872-9681
• DOI: 10.1016/j.asoc.2015.06.041

•Simulation-based learning of an optimal control policy for an artificial pancreas.•Integration of reinforcement learning with Gaussian processes for policy iteration.•Responding promptly to the varying activity levels seen in outpatients.•Stochastic modeling of diabetic patients for controlling variability. Automated control of blood glucose (BG) concentration with a fully automated artificial pancreas will certainly improve the quality of life for insulin-dependent patients. Closed-loop insulin delivery is challenging due to inter- and intra-patient variability, errors in glucose sensors and delays in insulin absorption. Responding to the varying activity levels seen in outpatients, with unpredictable and unreported food intake, and providing the necessary personalized control for individuals is a challenging task for existing control algorithms. A novel approach for controlling glycemic variability using simulation-based learning is presented. A policy iteration algorithm that combines reinforcement learning with Gaussian process approximation is proposed. To account for multiple sources of uncertainty, a control policy is learned off-line using an Ito's stochastic model of the glucose-insulin dynamics. For safety and performance, only relevant data are sampled through Bayesian active learning. Results obtained demonstrate that a generic policy is both safe and efficient for controlling subject-specific variability due to a patient's lifestyle and its distinctive metabolic response.