Character Recognition in Air-Writing Based on Network of Radars for Human-Machine Interface

• Published in: IEEE sensors journal Vol. 19; no. 19; pp. 8855 - 8864
• Main Authors: Arsalan, Muhammad, Santra, Avik
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.10.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 60 GHz mm-wave radar; Algorithms; Artificial neural networks; Character recognition; Chirp; connectionist temporal classification; convolutional LSTM; deep convolutional neural network; gesture recognition; Handwriting; Handwriting recognition; human-machine interface; Image classification; long short term memory; Man-machine interfaces; Millimeter waves; network of radars; Radar; Radar antennas; Radar cross-sections; sensing; Sensors; Trajectories; Trajectory; Writing
• ISSN: 1530-437X; 1558-1748
• DOI: 10.1109/JSEN.2019.2922395

Radar technology plays a vital role in contact-less detection of hand gestures or motions, which forms an alternate and intuitive form of human-computer interface. Air-writing refers to the writing of linguistic characters or words in free space by hand gesture movements. In this paper, we propose an air-writing system based on a network of millimeter wave radars. We propose a two-stage approach for extraction and recognition of handwriting gestures. The extraction processing stage uses a fine range estimate combined with the trilateration technique to detect and localize the hand marker, followed by a smoothening filter to create a trajectory of the character through the hand movement. For the recognition stage, we explore two approaches: one extracts the time-series trajectory data and recognizes the drawn character using long short term memory (LSTM), bi-directional LSTM (BLSTM), and convolutional LSTM (ConvLSTM) with connectionist temporal classification (CTC) loss function, and the other approach reconstructs a 2D image from the trajectory of drawn character and uses deep convolutional neural network (DCNN) to classify the alphabets drawn by the user. ConvLSTM-CTC performs best among LSTM variants on time-series trajectory data achieving 98.33% classification accuracy similar to DCNN over the chosen character set. This paper employs real data using a network of three 60-GHz millimeter wave radar sensor to demonstrate the success of the proposed setup and associated algorithm with design consideration.