Learning Deep Structure-Preserving Image-Text Embeddings

This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a large-margin objective that combines cross-view ranking constraints with within-view n...

Full description

Saved in:
Bibliographic Details
Published in2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 5005 - 5013
Main Authors Liwei Wang, Yin Li, Lazebnik, Svetlana
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.06.2016
Subjects
Bidirectional control
Computer vision
Image color analysis
Kernel
Machine learning
Training
Visualization
Online AccessGet full text

Cover

More Information
Summary:This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a large-margin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and text-to-image retrieval. Our method achieves new state-of-the-art results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.
ISSN:1063-6919
DOI:10.1109/CVPR.2016.541