Addressing Predicate Overlap in Scene Graph Generation with Semantic Granularity Controller

• Published in: Proceedings (IEEE International Conference on Multimedia and Expo) pp. 78 - 83
• Main Authors: Chen, Guikun, Li, Lin, Luo, Yawei, Xiao, Jun
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.07.2023
• Subjects: Annotations; Benchmark testing; Label confusion; Partial multi-label learning; Predicate overlap; Predictive models; Scene graph generation; Semantics; Task analysis; Training
• ISSN: 1945-788X
• DOI: 10.1109/ICME55011.2023.00022

Semantic overlap between predicates (e.g., riding versus on) occurs inevitably when describing a scene. However, most existing Scene Graph Generation (SGG) works sidestep it by modeling the semantic overlap at category-level and assigning merely one-hot target to each sample, which hurt the performance on other reasonable predicates. In this paper, we argue that semantic overlap between predicates tends to vary in different abstract patterns, and a subject-object pair should retain multiple reasonable predicates. To this end, we make an early attempt to reformulate SGG as a partial multi-label learning problem and accordingly propose a model-agnostic Semantic Granularity Controller (SGC). SGC consists of a pattern-specific controller, partial multi-label learning, and controllable inference. The former two solve semantic confusion during training, while the latter makes the semantic granularity of prediction controllable. Extensive experiments demonstrate that SGC can improve the performance of SGG and guide the model to predict coarse/fine-grained predicates.