Automatic Judgement of Neural Network-Generated Image Captions

• Published in: Statistical Language and Speech Processing pp. 261 - 272
• Main Authors: Biswas, Rajarshi, Mogadala, Aditya, Barz, Michael, Sonntag, Daniel, Klakow, Dietrich
• Format: Book Chapter
• Language: English
• Published: Cham Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Active learning; Automated human judgement; Image captioning; Neural networks; NLG; NLP
• ISBN: 9783030313715; 3030313719
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-31372-2_22

Manual evaluation of individual results of natural language generation tasks is one of the bottlenecks. It is very time consuming and expensive if it is, for example, crowdsourced. In this work, we address this problem for the specific task of automatic image captioning. We automatically generate human-like judgements on grammatical correctness, image relevance and diversity of the captions obtained from a neural image caption generator. For this purpose, we use pool-based active learning with uncertainty sampling and represent the captions using fixed size vectors from Google’s Universal Sentence Encoder. In addition, we test common metrics, such as BLEU, ROUGE, METEOR, Levenshtein distance, and n-gram counts and report F1 score for the classifiers used under the active learning scheme for this task. To the best of our knowledge, our work is the first in this direction and promises to reduce time, cost, and human effort.