Audiovisual integration of emotional signals in voice and face: An event-related fMRI study

• Published in: NeuroImage (Orlando, Fla.) Vol. 37; no. 4; pp. 1445 - 1456
• Main Authors: Kreifelts, Benjamin, Ethofer, Thomas, Grodd, Wolfgang, Erb, Michael, Wildgruber, Dirk
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.10.2007; Elsevier Limited
• Subjects: Adult; Audiovisual; Brain; Cerebrovascular Circulation - physiology; Classification; Effective connectivity; Electrophysiology; Emotion; Emotions - physiology; Evoked Potentials - physiology; Facial Expression; Female; Functional Laterality - physiology; Fusiform; Humans; Image Processing, Computer-Assisted; Integration; Magnetic Resonance Imaging; Male; Neural networks; Oxygen - blood; Photic Stimulation; Prosody; Social Perception; STS; Studies; Temporal Lobe - blood supply; Temporal Lobe - physiology; Thalamus - blood supply; Thalamus - physiology; Voice; Audiovisual; Integration; STS; Prosody; Fusiform; Emotion; Facial expression; Effective connectivity
• ISSN: 1053-8119; 1095-9572
• DOI: 10.1016/j.neuroimage.2007.06.020

In a natural environment, non-verbal emotional communication is multimodal (i.e. speech melody, facial expression) and multifaceted concerning the variety of expressed emotions. Understanding these communicative signals and integrating them into a common percept is paramount to successful social behaviour. While many previous studies have focused on the neurobiology of emotional communication in the auditory or visual modality alone, far less is known about multimodal integration of auditory and visual non-verbal emotional information. The present study investigated this process using event-related fMRI. Behavioural data revealed that audiovisual presentation of non-verbal emotional information resulted in a significant increase in correctly classified stimuli when compared with visual and auditory stimulation. This behavioural gain was paralleled by enhanced activation in bilateral posterior superior temporal gyrus (pSTG) and right thalamus, when contrasting audiovisual to auditory and visual conditions. Further, a characteristic of these brain regions, substantiating their role in the emotional integration process, is a linear relationship between the gain in classification accuracy and the strength of the BOLD response during the bimodal condition. Additionally, enhanced effective connectivity between audiovisual integration areas and associative auditory and visual cortices was observed during audiovisual stimulation, offering further insight into the neural process accomplishing multimodal integration. Finally, we were able to document an enhanced sensitivity of the putative integration sites to stimuli with emotional non-verbal content as compared to neutral stimuli.