STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos

• Published in: arXiv.org
• Main Authors: Athar, Ali, Mahadevan, Sabarinath, Ošep, Aljoša, Leal-Taixé, Laura, Leibe, Bastian
• Format: Paper Journal Article
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 01.09.2023
• Subjects: Clusters; Computer Science - Computer Vision and Pattern Recognition; Computer Science - Learning; Image detection; Image segmentation; Instance segmentation; Object recognition
• ISSN: 2331-8422
• DOI: 10.48550/arxiv.2003.08429

Existing methods for instance segmentation in videos typically involve multi-stage pipelines that follow the tracking-by-detection paradigm and model a video clip as a sequence of images. Multiple networks are used to detect objects in individual frames, and then associate these detections over time. Hence, these methods are often non-end-to-end trainable and highly tailored to specific tasks. In this paper, we propose a different approach that is well-suited to a variety of tasks involving instance segmentation in videos. In particular, we model a video clip as a single 3D spatio-temporal volume, and propose a novel approach that segments and tracks instances across space and time in a single stage. Our problem formulation is centered around the idea of spatio-temporal embeddings which are trained to cluster pixels belonging to a specific object instance over an entire video clip. To this end, we introduce (i) novel mixing functions that enhance the feature representation of spatio-temporal embeddings, and (ii) a single-stage, proposal-free network that can reason about temporal context. Our network is trained end-to-end to learn spatio-temporal embeddings as well as parameters required to cluster these embeddings, thus simplifying inference. Our method achieves state-of-the-art results across multiple datasets and tasks. Code and models are available at https://github.com/sabarim/STEm-Seg.