Object Segmentation from Open-Vocabulary Manipulation Instructions Based on Optimal Transport Polygon Matching with Multimodal Foundation Models

We consider the task of generating segmentation masks for the target object from an object manipulation instruction, which allows users to give open vocabulary instructions to domestic service robots. Conventional segmentation generation approaches often fail to account for objects outside the camer...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Nishimura, Takayuki, Kuyo, Katsuyuki, Kambara, Motonari, Sugiura, Komei
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 01.07.2024
Subjects
Apexes
Datasets
Effectiveness
Masks
Polygons
Segmentation
Service robots
Target masking
Online AccessGet full text

Cover

More Information
Summary:We consider the task of generating segmentation masks for the target object from an object manipulation instruction, which allows users to give open vocabulary instructions to domestic service robots. Conventional segmentation generation approaches often fail to account for objects outside the camera's field of view and cases in which the order of vertices differs but still represents the same polygon, which leads to erroneous mask generation. In this study, we propose a novel method that generates segmentation masks from open vocabulary instructions. We implement a novel loss function using optimal transport to prevent significant loss where the order of vertices differs but still represents the same polygon. To evaluate our approach, we constructed a new dataset based on the REVERIE dataset and Matterport3D dataset. The results demonstrated the effectiveness of the proposed method compared with existing mask generation methods. Remarkably, our best model achieved a +16.32% improvement on the dataset compared with a representative polygon-based method.
ISSN:2331-8422