Real Time Air-Written Mathematical Expression Recognition for Children’s Enhanced Learning

• Published in: Neural processing letters Vol. 55; no. 3; pp. 3355 - 3375
• Main Authors: Kumar, Shobhan, Chandra Trivedi, Munesh, Chauhan, Arun
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.06.2023; Springer Nature B.V
• Subjects: Artificial Intelligence; Artificial neural networks; Complex Systems; Computational Intelligence; Computer Science; Datasets; Deep learning; Fingers; Handwriting; Keyboards; Mathematical analysis; Operators (mathematics); Recognition; Robustness (mathematics); Sensors; Support vector machines; Webcams; Writing; Convolutional neural network (CNN); Air writing; Air-written character recognition; Human- computer interaction (HCI)
• ISSN: 1370-4621; 1573-773X
• DOI: 10.1007/s11063-022-11012-3

Air-writing is the process where, without the assistance of any handheld device, users use finger or hand gestures to write a character or words in free space. Due to its simple writing style, it has a great advantage over conventional pen-and-paper-based systems. However, because of the absence of any common delimiting criterion, non-uniform characters, and different writing styles, it is a difficult task. In this work, we propose an air written Mathematical expression recognition system using webcam video as input. We employed a new hand detection model that recognizes the writing hand and tracks the fingertip movement to collect the trajectories and then the convolutional neural network (CNN) is used as a recognizer. Through our model children can explore basic ME evaluation on their own without any instructor’s help and can gain more knowledge with minimal effort. Experiments were conducted on a combination of MNIST, ISI Air-Written English numerals, RTD along our own air-written Math operators datasets (MAIR). To evaluate the robustness of our model, we also tested our model on a group of children where they fed the input by writing in the air and the input data was captured using a system webcam. In both cases, we achieved promising results for digit recognition as well as ME evaluation.