Empirical Analysis of Image Caption Generation using Deep Learning

Automated image captioning is one of the applications of Deep Learning which involves fusion of work done in computer vision and natural language processing, and it is typically performed using Encoder-Decoder architectures. In this project, we have implemented and experimented with various flavors...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Bhattacharya, Aditya, Girishekar, Eshwar Shamanna, Padmakar Anil Deshpande
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 22.05.2021
Subjects
Cider
Coders
Computer vision
Deep learning
Empirical analysis
Encoders-Decoders
Natural language processing
Online AccessGet full text

Cover

More Information
Summary:Automated image captioning is one of the applications of Deep Learning which involves fusion of work done in computer vision and natural language processing, and it is typically performed using Encoder-Decoder architectures. In this project, we have implemented and experimented with various flavors of multi-modal image captioning networks where ResNet101, DenseNet121 and VGG19 based CNN Encoders and Attention based LSTM Decoders were explored. We have studied the effect of beam size and the use of pretrained word embeddings and compared them to baseline CNN encoder and RNN decoder architecture. The goal is to analyze the performance of each approach using various evaluation metrics including BLEU, CIDEr, ROUGE and METEOR. We have also explored model explainability using Visual Attention Maps (VAM) to highlight parts of the images which has maximum contribution for predicting each word of the generated caption.
ISSN:2331-8422