Wearable on-device deep learning system for hand gesture recognition based on FPGA accelerator

• Published in: Mathematical biosciences and engineering : MBE Vol. 18; no. 1; pp. 132 - 153
• Main Authors: Jiang, Weibin, Ye, Xuelin, Chen, Ruiqi, Su, Feng, Lin, Mengru, Ma, Yuhanxiao, Zhu, Yanxiang, Huang, Shizhen
• Format: Journal Article
• Language: English
• Published: United States AIMS Press 01.01.2021
• Subjects: accelerator; convolutional neural network (cnn); field-programmable gate array (fpga); gesture recognition; inertial measurement unit (imu); micro-control unit (mcu); accelerator; inertial measurement unit (IMU); gesture recognition; convolutional neural network (CNN); field-programmable gate array (FPGA); micro-control unit (MCU)
• ISSN: 1551-0018; 1551-0018
• DOI: 10.3934/mbe.2021007

Gesture recognition is critical in the field of Human-Computer Interaction, especially in healthcare, rehabilitation, sign language translation, etc. Conventionally, the gesture recognition data collected by the inertial measurement unit (IMU) sensors is relayed to the cloud or a remote device with higher computing power to train models. However, it is not convenient for remote follow-up treatment of movement rehabilitation training. In this paper, based on a field-programmable gate array (FPGA) accelerator and the Cortex-M0 IP core, we propose a wearable deep learning system that is capable of locally processing data on the end device. With a pre-stage processing module and serial-parallel hybrid method, the device is of low-power and low-latency at the micro control unit (MCU) level, however, it meets or exceeds the performance of single board computers (SBC). For example, its performance is more than twice as much of Cortex-A53 (which is usually used in Raspberry Pi). Moreover, a convolutional neural network (CNN) and a multilayer perceptron neural network (NN) is used in the recognition model to extract features and classify gestures, which helps achieve a high recognition accuracy at 97%. Finally, this paper offers a software-hardware co-design method that is worth referencing for the design of edge devices in other scenarios.