PseudoClick: Interactive Image Segmentation with Click Imitation

• Published in: Computer Vision - ECCV 2022 Vol. 13666; pp. 728 - 745
• Main Authors: Liu, Qin, Zheng, Meng, Planche, Benjamin, Karanam, Srikrishna, Chen, Terrence, Niethammer, Marc, Wu, Ziyan
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer 2022; Springer Nature Switzerland
• Series: Lecture Notes in Computer Science
• Subjects: Click imitation; Interactive image segmentation; Pseudo click
• ISBN: 9783031200670; 3031200675
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-031-20068-7_42

The goal of click-based interactive image segmentation is to obtain precise object segmentation masks with limited user interaction, i.e., by a minimal number of user clicks. Existing methods require users to provide all the clicks: by first inspecting the segmentation mask and then providing points on mislabeled regions, iteratively. We ask the question: can our model directly predict where to click, so as to further reduce the user interaction cost? To this end, we propose PseudoClick, a generic framework that enables existing segmentation networks to propose candidate next clicks. These automatically generated clicks, termed pseudo clicks in this work, serve as an imitation of human clicks to refine the segmentation mask. We build PseudoClick on existing segmentation backbones and show how the click prediction mechanism leads to improved performance. We evaluate PseudoClick on 10 public datasets from different domains and modalities, showing that our model not only outperforms existing approaches but also demonstrates strong generalization capability in cross-domain evaluation. We obtain new state-of-the-art results on several popular benchmarks, e.g., on the Pascal dataset, our model significantly outperforms existing state-of-the-art by reducing 12.4% number of clicks to achieve 85% IoU.