Vocabulary Learning Support System Based on Automatic Image Captioning Technology

• Published in: Distributed, Ambient and Pervasive Interactions Vol. 11587; pp. 346 - 358
• Main Authors: Hasnine, Mohammad Nehal, Flanagan, Brendan, Akcapinar, Gokhan, Ogata, Hiroaki, Mouri, Kousuke, Uosaki, Noriko
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2019; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Artificial intelligence in education; Automatic image captioning; Learning context representation; Ubiquitous learning; Visual contents analysis; Vocabulary learning
• ISBN: 9783030219345; 3030219348
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-21935-2_26

Learning context has evident to be an essential part in vocabulary development, however describing learning context for each vocabulary is considered to be difficult. In the human brain, it is relatively easy to describe learning contexts using pictures because pictures describe an immense amount of details at a quick glance that text annotations cannot do. Therefore, in an informal language learning system, pictures can be used to overcome the problems that language learners face in describing learning contexts. The present study aimed to develop a support system that generates and represents learning contexts automatically by analyzing the visual contents of the pictures captured by language learners. Automatic image captioning, a technology of artificial intelligence that connects computer vision and natural language processing is used for analyzing the visual contents of the learners’ captured images. A neural image caption generator model called Show and Tell is trained for image-to-word generation and to describe the context of an image. The three-fold objectives of this research are: First, an intelligent technology that can understand the contents of the picture and capable to generate learning contexts automatically; Second, a leaner can learn multiple vocabularies by using one picture without relying on a representative picture for each vocabulary, and Third, a learner’s prior vocabulary knowledge can be mapped with new learning vocabulary so that previously acquired vocabulary be reviewed and recalled while learning new vocabulary.