On-line Integration of Semantic Information from Speech and Gesture: Insights from Event-related Brain Potentials

• Published in: Journal of cognitive neuroscience Vol. 19; no. 4; pp. 605 - 616
• Main Authors: Özyürek, Aslı, Willems, Roel M., Kita, Sotaro, Hagoort, Peter
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.04.2007; MIT Press Journals, The
• Subjects: Adult; Behavioral psychophysiology; Biological and medical sciences; Brain; Brain research; Cognition & reasoning; Comprehension; Comprehension - physiology; Electrophysiology; Evoked Potentials - physiology; Evoked Responses; Female; Fundamental and applied biological sciences. Psychology; Gestures; Humans; Iconicity; Integration; Language; Language Tests; Male; Neurosciences; Nonverbal Communication; Production and perception of spoken language; Psychology. Psychoanalysis. Psychiatry; Psychology. Psychophysiology; Semantic Processing; Semantics; Speech Perception - physiology; Visual Perception - physiology; Human; Verbal comprehension; Information integration; Central nervous system; Electrophysiology; Cognition; Encephalon; Language; Semantics; Gesture; Insight; Speech; Event evoked potential
• ISSN: 0898-929X; 1530-8898
• DOI: 10.1162/jocn.2007.19.4.605

During language comprehension, listeners use the global semantic representation from previous sentence or discourse context to immediately integrate the meaning of each upcoming word into the unfolding message-level representation. Here we investigate whether communicative gestures that often spontaneously co-occur with speech are processed in a similar fashion and integrated to previous sentence context in the same way as lexical meaning. Event-related potentials were measured while subjects listened to spoken sentences with a critical verb (e.g., ), which was accompanied by an iconic co-speech gesture (i.e., KNOCK). Verbal and/or gestural semantic content matched or mismatched the content of the preceding part of the sentence. Despite the difference in the modality and in the specificity of meaning conveyed by spoken words and gestures, the latency, amplitude, and topographical distribution of both word and gesture mismatches are found to be similar, indicating that the brain integrates both types of information simultaneously. This provides evidence for the claim that neural processing in language comprehension involves the simultaneous incorporation of information coming from a broader domain of cognition than only verbal semantics. The neural evidence for similar integration of information from speech and gesture emphasizes the tight interconnection between speech and co-speech gestures.