Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities

• Published in: Brain sciences Vol. 10; no. 7; p. 442
• Main Authors: Wang, You, Zhang, Ming, Wu, RuMeng, Gao, Han, Yang, Meng, Luo, Zhiyuan, Li, Guang
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 01.07.2020; MDPI
• Subjects: bidirectional long short-term memory; Cellular signal transduction; Design and construction; Human-machine systems; neuromuscular signal; Neuromuscular transmission; Observations; Physiological aspects; silent speech decoding; spectrogram features; Spectrographs; Speech; User interfaces (Computers); Xception; China; Xception; neuromuscular signal; spectrogram features; silent speech decoding; bidirectional long short-term memory
• ISSN: 2076-3425; 2076-3425
• DOI: 10.3390/brainsci10070442

Silent speech decoding is a novel application of the Brain–Computer Interface (BCI) based on articulatory neuromuscular activities, reducing difficulties in data acquirement and processing. In this paper, spatial features and decoders that can be used to recognize the neuromuscular signals are investigated. Surface electromyography (sEMG) data are recorded from human subjects in mimed speech situations. Specifically, we propose to utilize transfer learning and deep learning methods by transforming the sEMG data into spectrograms that contain abundant information in time and frequency domains and are regarded as channel-interactive. For transfer learning, a pre-trained model of Xception on the large image dataset is used for feature generation. Three deep learning methods, Multi-Layer Perception, Convolutional Neural Network and bidirectional Long Short-Term Memory, are then trained using the extracted features and evaluated for recognizing the articulatory muscles’ movements in our word set. The proposed decoders successfully recognized the silent speech and bidirectional Long Short-Term Memory achieved the best accuracy of 90%, outperforming the other two algorithms. Experimental results demonstrate the validity of spectrogram features and deep learning algorithms.