Long short‐term memory‐based neural decoding of object categories evoked by natural images

• Published in: Human brain mapping Vol. 41; no. 15; pp. 4442 - 4453
• Main Authors: Huang, Wei, Yan, Hongmei, Wang, Chong, Li, Jiyi, Yang, Xiaoqing, Li, Liang, Zuo, Zhentao, Zhang, Jiang, Chen, Huafu
• Format: Journal Article
• Language: English
• Published: Hoboken, USA John Wiley & Sons, Inc 15.10.2020
• Subjects: Adult; Brain Mapping; Categories; Cognitive ability; Comparative analysis; Computer simulation; Concept Formation - physiology; deep learning; Female; fMRI; Functional magnetic resonance imaging; Humans; LSTM; Magnetic Resonance Imaging; Male; Mapping; Memory; Models, Theoretical; Nerve Net - diagnostic imaging; Nerve Net - physiology; Nervous system; Neural coding; Neural Networks, Computer; Neuroimaging; Pattern Recognition, Visual - physiology; Signal processing; Time dependence; Visual cortex; Visual Cortex - physiology; visual perceptual decoding; Visual signals; Young Adult; visual perceptual decoding; deep learning; fMRI; LSTM; visual cortex
• ISSN: 1065-9471; 1097-0193; 1097-0193
• DOI: 10.1002/hbm.25136

Visual perceptual decoding is one of the important and challenging topics in cognitive neuroscience. Building a mapping model between visual response signals and visual contents is the key point of decoding. Most previous studies used peak response signals to decode object categories. However, brain activities measured by functional magnetic resonance imaging are a dynamic process with time dependence, so peak signals cannot fully represent the whole process, which may affect the performance of decoding. Here, we propose a decoding model based on long short‐term memory (LSTM) network to decode five object categories from multitime response signals evoked by natural images. Experimental results show that the average decoding accuracy using the multitime (2–6 s) response signals is 0.540 from the five subjects, which is significantly higher than that using the peak ones (6 s; accuracy: 0.492; p < .05). In addition, from the perspective of different durations, methods and visual areas, the decoding performances of the five object categories are deeply and comprehensively explored. The analysis of different durations and decoding methods reveals that the LSTM‐based decoding model with sequence simulation ability can fit the time dependence of the multitime visual response signals to achieve higher decoding performance. The comparative analysis of different visual areas demonstrates that the higher visual cortex (VC) contains more semantic category information needed for visual perceptual decoding than lower VC. Visual perceptual decoding is one of the important and challenging topics in cognitive neuroscience. Building a mapping model between visual response signals and visual contents is the key point of decoding.