Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation

• Published in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 4101 - 4110
• Main Authors: Zhang, Zhenyu, Cui, Zhen, Xu, Chunyan, Yan, Yan, Sebe, Nicu, Yang, Jian
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2019
• Subjects: 3D from Single Image; Segmentation; Grouping and Shape; Scene Analysis and Understanding
• ISSN: 2575-7075
• DOI: 10.1109/CVPR.2019.00423

In this paper, we propose a novel Pattern-Affinitive Propagation (PAP) framework to jointly predict depth, surface normal and semantic segmentation. The motivation behind it comes from the statistic observation that pattern-affinitive pairs recur much frequently across different tasks as well as within a task. Thus, we can conduct two types of propagations, cross-task propagation and task-specific propagation, to adaptively diffuse those similar patterns. The former integrates cross-task affinity patterns to adapt to each task therein through the calculation on non-local relationships. Next the latter performs an iterative diffusion in the feature space so that the cross-task affinity patterns can be widely-spread within the task. Accordingly, the learning of each task can be regularized and boosted by the complementary task-level affinities. Extensive experiments demonstrate the effectiveness and the superiority of our method on the joint three tasks. Meanwhile, we achieve the state-of-the-art or competitive results on the three related datasets, NYUD-v2, SUN-RGBD and KITTI.