Verbal working memory and co-speech gesture processing

• Published in: Brain and cognition Vol. 146; p. 105640
• Main Authors: Momsen, Jacob, Gordon, Jared, Wu, Ying Choon, Coulson, Seana
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.12.2020; Elsevier Science
• Subjects: Cognitive ability; Comprehension; Event-related potentials; Gestures; Humans; Iconic gestures; Integration; Language; Memory; Memory, Short-Term; Mental Processes; Mental task performance; Multisensory integration; Representational gestures; Resource availability; Short term memory; Speech; Speech Perception; Speech-gesture integration; Working memory; Multisensory integration; Working memory; Iconic gestures; Representational gestures; Speech-gesture integration
• ISSN: 0278-2626; 1090-2147; 1090-2147
• DOI: 10.1016/j.bandc.2020.105640

•EEG recorded to speech with semantically congruent/incongruent gestures under high/low verbal load.•ERP to words with congruent gestures not impacted by VWM load suggesting automaticity.•Words with incongruent gestures elicit enhanced N400 with low load, P3a with high load.•Speech-gesture integration similar to basic sensory integration in use of top-down attention.•VWM recruited for speech-gesture integration when meaning of gestures is unclear. Multimodal discourse requires an assembly of cognitive processes that are uniquely recruited for language comprehension in social contexts. In this study, we investigated the role of verbal working memory for the online integration of speech and iconic gestures. Participants memorized and rehearsed a series of auditorily presented digits in low (one digit) or high (four digits) memory load conditions. To observe how verbal working memory load impacts online discourse comprehension, ERPs were recorded while participants watched discourse videos containing either congruent or incongruent speech-gesture combinations during the maintenance portion of the memory task. While expected speech-gesture congruity effects were found in the low memory load condition, high memory load trials elicited enhanced frontal positivities that indicated a unique interaction between online speech-gesture integration and the availability of verbal working memory resources. This work contributes to an understanding of discourse comprehension by demonstrating that language processing in a multimodal context is subject to the relationship between cognitive resource availability and the degree of controlled processing required for task performance. We suggest that verbal working memory is less important for speech-gesture integration than it is for mediating speech processing under high task demands.