Semantic reconstruction of continuous language from non-invasive brain recordings

• Published in: Nature neuroscience Vol. 26; no. 5; pp. 858 - 866
• Main Authors: Tang, Jerry, LeBel, Amanda, Jain, Shailee, Huth, Alexander G.
• Format: Journal Article
• Language: English
• Published: New York Nature Publishing Group US 01.05.2023; Nature Publishing Group
• Subjects: 59/36; 59/57; 631/1647/245/1627; 631/378/116/2394; 631/378/2649/1594; Animal Genetics and Genomics; Behavioral Sciences; Biological Techniques; Biomedical and Life Sciences; Biomedicine; Brain; Brain mapping; Brain Mapping - methods; Brain-Computer Interfaces; Cooperation; Decoders; Decoding; Functional magnetic resonance imaging; Image reconstruction; Interfaces; Language; Magnetic resonance imaging; Magnetic Resonance Imaging - methods; Neurobiology; Neuroimaging; Neurosciences; Semantics; Speech; Speech Perception; Stimuli; Video
• ISSN: 1097-6256; 1546-1726; 1546-1726
• DOI: 10.1038/s41593-023-01304-9

A brain–computer interface that decodes continuous language from non-invasive recordings would have many scientific and practical applications. Currently, however, non-invasive language decoders can only identify stimuli from among a small set of words or phrases. Here we introduce a non-invasive decoder that reconstructs continuous language from cortical semantic representations recorded using functional magnetic resonance imaging (fMRI). Given novel brain recordings, this decoder generates intelligible word sequences that recover the meaning of perceived speech, imagined speech and even silent videos, demonstrating that a single decoder can be applied to a range of tasks. We tested the decoder across cortex and found that continuous language can be separately decoded from multiple regions. As brain–computer interfaces should respect mental privacy, we tested whether successful decoding requires subject cooperation and found that subject cooperation is required both to train and to apply the decoder. Our findings demonstrate the viability of non-invasive language brain–computer interfaces. Tang et al. show that continuous language can be decoded from functional MRI recordings to recover the meaning of perceived and imagined speech stimuli and silent videos and that this language decoding requires subject cooperation.