Context-Aware Group Captioning via Self-Attention and Contrastive Features

While image captioning has progressed rapidly, existing works focus mainly on describing single images. In this paper, we introduce a new task, context-aware group captioning, which aims to describe a group of target images in the context of another group of related reference images. Context-aware g...

Full description

Saved in:
Bibliographic Details
Published in2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3437 - 3447
Main Authors Li, Zhuowan, Tran, Quan, Mai, Long, Lin, Zhe, Yuille, Alan L.
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.01.2020
Subjects
Computational modeling
Computer vision
Context modeling
Natural languages
Task analysis
Training
Visualization
Online AccessGet full text

Cover

More Information
Summary:While image captioning has progressed rapidly, existing works focus mainly on describing single images. In this paper, we introduce a new task, context-aware group captioning, which aims to describe a group of target images in the context of another group of related reference images. Context-aware group captioning requires not only summarizing information from both the target and reference image group but also contrasting between them. To solve this problem, we propose a framework combining self-attention mechanism with contrastive feature construction to effectively summarize common information from each image group while capturing discriminative information between them. To build the dataset for this task, we propose to group the images and generate the group captions based on single image captions using scene graphs matching. Our datasets are constructed on top of the public Conceptual Captions dataset and our new Stock Captions dataset. Experiments on the two datasets show the effectiveness of our method on this new task.
ISSN:2575-7075
DOI:10.1109/CVPR42600.2020.00350