Out-of-Distribution Semantic Segmentation with Disentangled and Calibrated Representation

• Published in: IEEE transactions on circuits and systems for video technology p. 1
• Main Authors: Wan, Maoxian, Li, Kaige, Geng, Qichuan, Su, Binyi, Zhou, Zhong
• Format: Journal Article
• Language: English
• Published: IEEE 2025
• Subjects: Accuracy; Data models; disentangled representations; Image reconstruction; out-of-distribution; Overfitting; Semantic segmentation; Shape; Synthetic data; Training; Training data; Uncertainty
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2025.3597071

Out-of-distribution (OoD) semantic segmentation aims to recognize pixels of classes undefined in the training dataset. Existing methods mostly focus on training the model to fit real OoD data samples to identify OoD pixels, which requires extra data collection and annotation efforts. By contrast, synthesizing OoD data with training data provides a more resource-efficient alternative. However, synthetic data generated from controlled settings lacks diversity, causing the model to suffer from overfitting. To this end, we propose a disentangled representation learning (DRL) method to guide the model to disentangle semantic-related and semantic-unrelated features from synthetic OoD data. DRL encourages the model to utilize the former to identify semantic categories, rather than overfitting to such semantic-unrelated features as synthetic artificiality. Specifically, DRL first incorporates two disentanglers to extract the semantic-related and -unrelated features and then applies a shuffle and reconstruction mechanism to regularize the disentangled features. Furthermore, to facilitate disentangling, we propose a pixel-wise feature similarity calibration (PSC) module, which utilizes more accurate ID-OoD similarity to calibrate inaccurate ID-OoD similarity learned exclusively from ID data. Thus, PSC delivers accurate and stable pixel-wise features for effective disentangling. Extensive experiments illustrate that the proposed method exhibits strong generalization ability. It attains 74.04% AuPRC and 20.82% FPR on Road Anomaly, 69.85% AuPRC and 5.78% FPR on Fishyscapes LostAndFound Validation Set, using SegFormer with the MiT-B5 backbone. Source code is available at https://github.com/WanMotion/DisentangledOoDSeg.