Neural correlates of multisensory reliability and perceptual weights emerge at early latencies during audio‐visual integration

• Published in: The European journal of neuroscience Vol. 46; no. 10; pp. 2565 - 2577
• Main Authors: Boyle, Stephanie C., Kayser, Stephanie J., Kayser, Christoph
• Format: Journal Article
• Language: English
• Published: France Wiley Subscription Services, Inc 01.11.2017; John Wiley and Sons Inc
• Subjects: Acoustic Stimulation; Adult; Auditory Perception - physiology; Brain - physiology; Choice Behavior; Cognitive Neuroscience; Component reliability; cross‐modal; Cues; Decoding; Discriminant Analysis; EEG; Electroencephalography; Female; Humans; Male; Neuroimaging; Neurology; Photic Stimulation; Psychophysics; Sensory integration; single‐trial analysis; Visual observation; Visual Perception - physiology; Young Adult; electroencephalography; cross-modal; psychophysics; single-trial analysis
• ISSN: 0953-816X; 1460-9568
• DOI: 10.1111/ejn.13724

To make accurate perceptual estimates, observers must take the reliability of sensory information into account. Despite many behavioural studies showing that subjects weight individual sensory cues in proportion to their reliabilities, it is still unclear when during a trial neuronal responses are modulated by the reliability of sensory information or when they reflect the perceptual weights attributed to each sensory input. We investigated these questions using a combination of psychophysics, EEG‐based neuroimaging and single‐trial decoding. Our results show that the weighted integration of sensory information in the brain is a dynamic process; effects of sensory reliability on task‐relevant EEG components were evident 84 ms after stimulus onset, while neural correlates of perceptual weights emerged 120 ms after stimulus onset. These neural processes had different underlying sources, arising from sensory and parietal regions, respectively. Together these results reveal the temporal dynamics of perceptual and neural audio‐visual integration and support the notion of temporally early and functionally specific multisensory processes in the brain. This study focuses on investigating the temporal evolution of reliability weighting during audio‐visual integration using EEG. Our results show that early effects (< 100 ms) of sensory reliability and perceptual weighting can be dissociated from the EEG signal and that these effects arise from early sensory and parietal regions, respectively. Together these results support the notion of temporally early and functionally specific multisensory processes in the brain.