ROCOv2: Radiology Objects in COntext Version 2, An Updated Multimodal Image Dataset

• Main Authors: Johannes Rückert, Louise Bloch, Raphael Brüngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Cynthia S. Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba Garcia Seco de Herrera, Henning Müller, Peter A. Horn, Felix Nensa, Christoph M. Friedrich
• Format: Data Set
• Language: English
• Published: Zenodo 10.11.2023
• Subjects: image captions; machine learning; medical concepts; radiology
• ISSN: 2052-4463; 2052-4463
• DOI: 10.5281/zenodo.8333644

Recent advances in deep learning techniques have enabled the development of systems for automatic analysis of medical images. These systems often require large amounts of training data with high quality labels, which is difficult and time consuming to generate. Here, we introduce Radiology Object in COntext Version 2 (ROCOv2), a multimodal dataset consisting of radiological images and associated medical concepts and captions extracted from the PubMed Open Access subset. Concepts for clinical modality, anatomy (X-ray), and directionality (X-ray) were manually curated and additionally evaluated by a radiologist. Unlike MIMIC-CXR, ROCOv2 includes seven different clinical modalities. It is an updated version of the ROCO dataset published in 2018, and includes 35,705 new images added to PubMed since 2018, as well as manually curated medical concepts for modality, body region (X-ray) and directionality (X-ray). The dataset consists of 79,789 images and has been used, with minor modifications, in the concept detection and caption prediction tasks of ImageCLEFmedical 2023. The participants had access to the training and validation sets after signing a user agreement. The dataset is suitable for training image annotation models based on image-caption pairs, or for multi-label image classification using the UMLS concepts provided with each image, e.g., to build systems to support structured medical reporting. Additional possible use cases for the ROCOv2 dataset include the pre-training of models for the medical domain, and the evaluation evaluation of deep learning models for multi-task learning.