Divide-and-Conquer Predictor for Unbiased Scene Graph Generation

• Published in: IEEE transactions on circuits and systems for video technology Vol. 32; no. 12; pp. 8611 - 8622
• Main Authors: Han, Xianjing, Dong, Xingning, Song, Xuemeng, Gan, Tian, Zhan, Yibing, Yan, Yan, Nie, Liqiang
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Bayes methods; Bayesian personalized ranking; Business process re-engineering; Classifiers; Correlation; Data mining; divide-and-conquer; Image analysis; Object recognition; Pattern recognition; Predictive models; Scene graph generation; Task analysis; vision and language
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2022.3193857

Scene Graph Generation (SGG) aims to detect the objects and their pairwise predicates in an image. Existing SGG methods mainly fulfil the challenging predicate prediction task that involves severe long-tailed data distribution with a single classifier. However, we argue that this may be enough to differentiate predicates that present obvious differences (e.g., <inline-formula> <tex-math notation="LaTeX">on </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">near </tex-math></inline-formula>), but not sufficient to distinguish similar predicates that only have subtle differences (e.g., <inline-formula> <tex-math notation="LaTeX">on </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">standing~on </tex-math></inline-formula>). Towards this end, we divide the predicate prediction into a few sub-tasks with a Divide-and-Conquer Predictor (DC-Predictor). Specifically, we first develop an offline pattern-predicate correlation mining algorithm to discover the similar predicates that share the same object interaction pattern. Based on that, we devise a general pattern classifier and a set of specific predicate classifiers for DC-Predictor. The former works on recognizing the pattern of a given object pair and routing it to the corresponding specific predicate classifier, while the latter aims to differentiate similar predicates in each specific pattern. In addition, we introduce the Bayesian Personalized Ranking loss in each specific predicate classifier to enhance the pairwise differentiation between head predicates and their similar ones. Experiments on VG150 and GQA datasets show the superiority of our model over state-of-the-art methods.