The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality

• Published in: The Journal of neuroscience Vol. 39; no. 39; pp. 7722 - 7736
• Main Authors: Deniz, Fatma, Nunez-Elizalde, Anwar O, Huth, Alexander G, Gallant, Jack L
• Format: Journal Article
• Language: English
• Published: United States Society for Neuroscience 25.09.2019
• Subjects: Alliances; Brain; Brain mapping; Cerebral cortex; Cortex (somatosensory); Functional magnetic resonance imaging; Language; Listening; Medical imaging; Neuroimaging; Neurology; Representations; Selectivity; Semantics; listening; fMRI; reading; BOLD; semantics; cross-modal representations
• ISSN: 0270-6474; 1529-2401
• DOI: 10.1523/jneurosci.0675-19.2019

An integral part of human language is the capacity to extract meaning from spoken and written words, but the precise relationship between brain representations of information perceived by listening versus reading is unclear. Prior neuroimaging studies have shown that semantic information in spoken language is represented in multiple regions in the human cerebral cortex, while amodal semantic information appears to be represented in a few broad brain regions. However, previous studies were too insensitive to determine whether semantic representations were shared at a fine level of detail rather than merely at a coarse scale. We used fMRI to record brain activity in two separate experiments while participants listened to or read several hours of the same narrative stories, and then created voxelwise encoding models to characterize semantic selectivity in each voxel and in each individual participant. We find that semantic tuning during listening and reading are highly correlated in most semantically selective regions of cortex, and models estimated using one modality accurately predict voxel responses in the other modality. These results suggest that the representation of language semantics is independent of the sensory modality through which the semantic information is received. Humans can comprehend the meaning of words from both spoken and written language. It is therefore important to understand the relationship between the brain representations of spoken or written text. Here, we show that although the representation of semantic information in the human brain is quite complex, the semantic representations evoked by listening versus reading are almost identical. These results suggest that the representation of language semantics is independent of the sensory modality through which the semantic information is received.