Picture-to-text Translation-based Assistive Communication System for People with Autism Spectrum Disorder

• Published in: Sensors and materials Vol. 34; no. 5; p. 2001
• Main Author: Chen, Yeou-Jiunn
• Format: Journal Article
• Language: English
• Published: Tokyo MYU Scientific Publishing Division 01.01.2022
• Subjects: Autism; Coders; Communications systems; Recurrent neural networks; Sequences
• ISSN: 0914-4935; 2435-0869
• DOI: 10.18494/SAM3773

People with autism spectrum disorder (ASD) have different degrees of communication disorder, reducing the quality of their daily lives. Therefore, an assistive communication system could effectively help people with ASD to communicate with other people or with devices. In this study, an assistive communication system with word prediction and picture-to-text conversion is proposed to help people with ASD. To help them use the system, touch sensors are adopted at the operating interfaces. To overcome the problem of the limited number of items that can be displayed by the operating interfaces, recurrent neural networks are adopted to predict the next input picture. To help people understand the meaning of the picture sequences inputted by people with ASD, a picture-to-text translation system, which is based on bidirectional encoder representations from transformers (BERTs), is integrated to convert picture sequences to sentences. Experimental results showed that the proposed assistive communication system can effectively predict the next picture and exactly convert picture sequences to sentences. Thus, the proposed system can effectively help people with ASD communicate with others.