OHO: A Multi-Modal, Multi-Purpose Dataset for Human-Robot Object Hand-Over

• Published in: Sensors (Basel, Switzerland) Vol. 23; no. 18; p. 7807
• Main Authors: Stephan, Benedict, Köhler, Mona, Müller, Steffen, Zhang, Yan, Gross, Horst-Michael, Notni, Gunther
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.09.2023; MDPI
• Subjects: 6D pose estimation; automated labeling; Automation; Cameras; Collaboration; dataset; Datasets; hand-over; Labeling; Machine learning; Robotics; Robotics industry; Robots; semantic segmentation; thermal image; Germany
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s23187807

In the context of collaborative robotics, handing over hand-held objects to a robot is a safety-critical task. Therefore, a robust distinction between human hands and presented objects in image data is essential to avoid contact with robotic grippers. To be able to develop machine learning methods for solving this problem, we created the OHO (Object Hand-Over) dataset of tools and other everyday objects being held by human hands. Our dataset consists of color, depth, and thermal images with the addition of pose and shape information about the objects in a real-world scenario. Although the focus of this paper is on instance segmentation, our dataset also enables training for different tasks such as 3D pose estimation or shape estimation of objects. For the instance segmentation task, we present a pipeline for automated label generation in point clouds, as well as image data. Through baseline experiments, we show that these labels are suitable for training an instance segmentation to distinguish hands from objects on a per-pixel basis. Moreover, we present qualitative results for applying our trained model in a real-world application.