Handwriting Recognition Based on 3D Accelerometer Data by Deep Learning

• Published in: Applied sciences Vol. 12; no. 13; p. 6707
• Main Authors: Lopez-Rodriguez, Pedro, Avina-Cervantes, Juan Gabriel, Contreras-Hernandez, Jose Luis, Correa, Rodrigo, Ruiz-Pinales, Jose
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.07.2022
• Subjects: 3D accelerometer data; 3D signal processing; Accelerometers; Algorithms; Classification; Convolutional Neural Networks (CNN); Deep learning; Discriminant analysis; Handwriting; handwritten character recognition; Interfaces; Long Short-Term Memory (LSTM); Neural networks; Sensors; Sign language; Smartphones; Smartwatches; Wearable computers; Writing
• ISSN: 2076-3417; 2076-3417
• DOI: 10.3390/app12136707

Online handwriting recognition has been the subject of research for many years. Despite that, a limited number of practical applications are currently available. The widespread use of devices such as smartphones, smartwatches, and tablets has not been enough to convince the user to use pen-based interfaces. This implies that more research on the pen interface and recognition methods is still necessary. This paper proposes a handwritten character recognition system based on 3D accelerometer signal processing using Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM). First, a user wearing an MYO armband on the forearm writes a multi-stroke freestyle character on a touchpad by using the finger or a pen. Next, the 3D accelerometer signals generated during the writing process are fed into a CNN, LSTM, or CNN-LSTM network for recognition. The convolutional backbone obtains spatial features in order to feed an LSTM that extracts short-term temporal information. The system was evaluated on a proprietary dataset of 3D accelerometer data collected from multiple users with an armband device, corresponding to handwritten English lowercase letters (a–z) and digits (0–9) in a freestyle. The results show that the proposed system overcomes other systems from the state of the art by 0.53%.