Neural signature of hierarchically structured expectations predicts clustering and transfer of rule sets in reinforcement learning

• Published in: Cognition Vol. 152; pp. 160 - 169
• Main Authors: Collins, Anne Gabrielle Eva, Frank, Michael Joshua
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 01.07.2016
• Subjects: Adolescent; Adult; Association Learning - physiology; Cerebral Cortex - physiology; Cluster Analysis; Clustering; EEG; Electroencephalography; Female; Humans; Male; Models, Psychological; Prefrontal cortex; Reinforcement (Psychology); Reward; Structure-learning; Transfer; Young Adult; Transfer; Structure-learning; Prefrontal cortex; Clustering; EEG
• ISSN: 0010-0277; 1873-7838
• DOI: 10.1016/j.cognition.2016.04.002

•Humans create latent hierarchical rule structure when learning from reinforcement.•They cluster such rules according to their popularity across contexts.•This allows immediate generalization of new information to equivalent contexts.•It also facilitates transfer of rules to new contexts.•EEG signals reflect hierarchically structured expectations and predict transfer. Often the world is structured such that distinct sensory contexts signify the same abstract rule set. Learning from feedback thus informs us not only about the value of stimulus-action associations but also about which rule set applies. Hierarchical clustering models suggest that learners discover structure in the environment, clustering distinct sensory events into a single latent rule set. Such structure enables a learner to transfer any newly acquired information to other contexts linked to the same rule set, and facilitates re-use of learned knowledge in novel contexts. Here, we show that humans exhibit this transfer, generalization and clustering during learning. Trial-by-trial model-based analysis of EEG signals revealed that subjects’ reward expectations incorporated this hierarchical structure; these structured neural signals were predictive of behavioral transfer and clustering. These results further our understanding of how humans learn and generalize flexibly by building abstract, behaviorally relevant representations of the complex, high-dimensional sensory environment.