TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. PP; no. 11; pp. 1 - 17
• Main Authors: Deng, Jiajun, Yang, Zhengyuan, Liu, Daqing, Chen, Tianlang, Zhou, Wengang, Zhang, Yanyong, Li, Houqiang, Ouyang, Wanli
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.11.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Coders; Cognition; Convolutional neural networks; Deep Learning; Grounding; Modules; Pipelines; Proposals; Transformer Network; Transformers; Vision; Vision and Language; Visual Grounding; Visualization
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2023.3296823

In this work, we explore neat yet effective Transformer-based frameworks for visual grounding. The previous methods generally address the core problem of visual grounding, i.e. , multi-modal fusion and reasoning, with manually-designed mechanisms. Such heuristic designs are not only complicated but also make models easily overfit specific data distributions. To avoid this, we first propose TransVG, which establishes multi-modal correspondences by Transformers and localizes referred regions by directly regressing box coordinates. We empirically show that complicated fusion modules can be replaced by a simple stack of Transformer encoder layers with higher performance. However, the core fusion Transformer in TransVG is stand-alone against uni-modal encoders, and thus should be trained from scratch on limited visual grounding data, which makes it hard to be optimized and leads to sub-optimal performance. To this end, we further introduce TransVG++ to make two-fold improvements. For one thing, we upgrade our framework to a purely Transformer-based one by leveraging Vision Transformer (ViT) for vision feature encoding. For another, we devise Language Conditioned Vision Transformer that removes external fusion modules and reuses the uni-modal ViT for vision-language fusion at the intermediate layers. We conduct extensive experiments on five prevalent datasets, and report a series of state-of-the-art records.