Effect of Kinematics and Fluency in Adversarial Synthetic Data Generation for ASL Recognition With RF Sensors

• Published in: IEEE transactions on aerospace and electronic systems Vol. 58; no. 4; pp. 2732 - 2745
• Main Authors: Rahman, Mohammad Mahbubur, Malaia, Evie A., Gurbuz, Ali Cafer, Griffin, Darrin J., Crawford, Chris, Gurbuz, Sevgi Zubeyde
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.08.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation; Adversarial learning; ASL; Auditory system; Deep learning; Doppler effect; Frequency modulation; Generative adversarial networks; Kinematics; Machine learning; micro-Doppler (<named-content xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" content-type="math" xlink:type="simple"> <inline-formula> <tex-math notation="LaTeX"> mu D</tex-math> </inline-formula> </named-content>); Natural language processing; radar; Radio frequency; Recognition; Sensors; sign language; Synthetic data; Training
• ISSN: 0018-9251; 1557-9603
• DOI: 10.1109/TAES.2021.3139848

RF sensors have been recently proposed as a new modality for sign language processing technology. They are noncontact, effective in the dark, and acquire a direct measurement of signing kinematic via exploitation of the micro-Doppler effect. First, this work provides an in depth comparative examination of the kinematic properties of signing as measured by RF sensors for both fluent ASL users and hearing imitation signers. Second, as ASL recognition techniques utilizing deep learning requires a large amount of training data, this work examines the effect of signing kinematics and subject fluency on adversarial learning techniques for data synthesis. The following two different approaches for the synthetic training data generation are proposed: 1) adversarial domain adaptation to minimize the differences between imitation signing and fluent signing data and 2) kinematically-constrained generative adversarial networks for accurate synthesis of RF signing signatures. The results show that the kinematic discrepancies between imitation signing and fluent signing are so significant that training on data directly synthesized from fluent RF signers offers greater performance (93% top-5 accuracy) than that produced by adaptation of imitation signing (88% top-5 accuracy) when classifying 100 ASL signs.