Neural decoding of attentional selection in multi-speaker environments without access to clean sources

• Published in: Journal of neural engineering Vol. 14; no. 5; p. 56001
• Main Authors: O'Sullivan, James, Chen, Zhuo, Herrero, Jose, McKhann, Guy M, Sheth, Sameer A, Mehta, Ashesh D, Mesgarani, Nima
• Format: Journal Article
• Language: English
• Published: England IOP Publishing 01.10.2017
• Subjects: Acoustic Stimulation - methods; attention; Auditory Cortex - physiology; Auditory Perception - physiology; DNN; ECoG; Electrodes, Implanted - trends; Electroencephalography - methods; Female; hearing aid; Hearing Aids - trends; Humans; LSTM; Male; Nerve Net - physiology; sEEG; Speech Perception - physiology; stimulus-reconstruction
• DOI: 10.1088/1741-2552/aa7ab4
• ISSN: 1741-2560; 1741-2552

Objective. People who suffer from hearing impairments can find it difficult to follow a conversation in a multi-speaker environment. Current hearing aids can suppress background noise; however, there is little that can be done to help a user attend to a single conversation amongst many without knowing which speaker the user is attending to. Cognitively controlled hearing aids that use auditory attention decoding (AAD) methods are the next step in offering help. Translating the successes in AAD research to real-world applications poses a number of challenges, including the lack of access to the clean sound sources in the environment with which to compare with the neural signals. We propose a novel framework that combines single-channel speech separation algorithms with AAD. Approach. We present an end-to-end system that (1) receives a single audio channel containing a mixture of speakers that is heard by a listener along with the listener's neural signals, (2) automatically separates the individual speakers in the mixture, (3) determines the attended speaker, and (4) amplifies the attended speaker's voice to assist the listener. Main results. Using invasive electrophysiology recordings, we identified the regions of the auditory cortex that contribute to AAD. Given appropriate electrode locations, our system is able to decode the attention of subjects and amplify the attended speaker using only the mixed audio. Our quality assessment of the modified audio demonstrates a significant improvement in both subjective and objective speech quality measures. Significance. Our novel framework for AAD bridges the gap between the most recent advancements in speech processing technologies and speech prosthesis research and moves us closer to the development of cognitively controlled hearable devices for the hearing impaired.