VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue

• Main Authors: Li, Yunshui, Hui, Binyuan, Yin, Zhaochao, He, Wanwei, Luo, Run, Long, Yuxing, Yang, Min, Huang, Fei, Li, Yongbin
• Format: Journal Article
• Language: English
• Published: 13.09.2023
• Subjects: Computer Science - Computation and Language; Computer Science - Computer Vision and Pattern Recognition
• DOI: 10.48550/arxiv.2309.07387

Visually-grounded dialog systems, which integrate multiple modes of communication such as text and visual inputs, have become an increasingly popular area of investigation. However, the absence of a standardized evaluation framework poses a challenge in assessing the development of this field. To this end, we propose \textbf{VDialogUE}, a \textbf{V}isually-grounded \textbf{Dialog}ue benchmark for \textbf{U}nified \textbf{E}valuation. It defines five core multi-modal dialogue tasks and covers six datasets. Furthermore, in order to provide a comprehensive assessment of the model's performance across all tasks, we developed a novel evaluation metric called VDscore, which is based on the Analytic Hierarchy Process~(AHP) method. Additionally, we present a straightforward yet efficient baseline model, named \textbf{VISIT}~(\textbf{VIS}ually-grounded d\textbf{I}alog \textbf{T}ransformer), to promote the advancement of general multi-modal dialogue systems. It progressively builds its multi-modal foundation and dialogue capability via a two-stage pre-training strategy. We believe that the VDialogUE benchmark, along with the evaluation scripts and our baseline models, will accelerate the development of visually-grounded dialog systems and lead to the development of more sophisticated and effective pre-trained models.