Instructional control of reinforcement learning: A behavioral and neurocomputational investigation

• Published in: Brain research Vol. 1299; pp. 74 - 94
• Main Authors: Doll, Bradley B., Jacobs, W. Jake, Sanfey, Alan G., Frank, Michael J.
• Format: Journal Article Conference Proceeding
• Language: English
• Published: Amsterdam Elsevier B.V 03.11.2009; Elsevier
• Subjects: Basal ganglia; Basal Ganglia - physiology; Biological and medical sciences; Choice Behavior - physiology; Computer Simulation; Dopamine; Dopamine - physiology; Fundamental and applied biological sciences. Psychology; Hippocampus - physiology; Humans; Models, Neurological; Nerve Net - physiology; Neural Networks, Computer; Neurology; Prefrontal Cortex - physiology; Psychomotor Performance - physiology; Reinforcement learning; Reinforcement, Psychology; Reward; Rule-governance; Vertebrates: nervous system and sense organs; Rule-governance; Basal ganglia; Dopamine; Reward; Reinforcement learning; Central nervous system; Rule-govemance; Basal ganglion; Catecholamine; Encephalon; Learning; Acquisition process; Behavioral analysis; Neurotransmitter; Reinforcement; Rule
• ISSN: 0006-8993; 1872-6240; 1872-6240
• DOI: 10.1016/j.brainres.2009.07.007

Humans learn how to behave directly through environmental experience and indirectly through rules and instructions. Behavior analytic research has shown that instructions can control behavior, even when such behavior leads to sub-optimal outcomes ( Hayes, S. (Ed.). 1989. Rule-governed behavior: cognition, contingencies, and instructional control. Plenum Press.). Here we examine the control of behavior through instructions in a reinforcement learning task known to depend on striatal dopaminergic function. Participants selected between probabilistically reinforced stimuli, and were (incorrectly) told that a specific stimulus had the highest (or lowest) reinforcement probability. Despite experience to the contrary, instructions drove choice behavior. We present neural network simulations that capture the interactions between instruction-driven and reinforcement-driven behavior via two potential neural circuits: one in which the striatum is inaccurately trained by instruction representations coming from prefrontal cortex/hippocampus (PFC/HC), and another in which the striatum learns the environmentally based reinforcement contingencies, but is “overridden” at decision output. Both models capture the core behavioral phenomena but, because they differ fundamentally on what is learned, make distinct predictions for subsequent behavioral and neuroimaging experiments. Finally, we attempt to distinguish between the proposed computational mechanisms governing instructed behavior by fitting a series of abstract “Q-learning” and Bayesian models to subject data. The best-fitting model supports one of the neural models, suggesting the existence of a “confirmation bias” in which the PFC/HC system trains the reinforcement system by amplifying outcomes that are consistent with instructions while diminishing inconsistent outcomes.