Configurable Graph Reasoning for Visual Relationship Detection

• Published in: IEEE transaction on neural networks and learning systems Vol. 33; no. 1; pp. 117 - 129
• Main Authors: Zhu, Yi, Liang, Xiwen, Lin, Bingqian, Ye, Qixiang, Jiao, Jianbin, Lin, Liang, Liang, Xiaodan
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Benchmarks; Cognition; Correlation; Feature extraction; Graph learning; Knowledge; Knowledge engineering; Knowledge representation; Labels; Neural Networks, Computer; Object recognition; Proposals; Reasoning; Recognition, Psychology; scene graph generation; Semantics; visual reasoning; visual relationship detection (VRD); Visualization
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2020.3027575

Visual commonsense knowledge has received growing attention in the reasoning of long-tailed visual relationships biased in terms of object and relation labels. Most current methods typically collect and utilize external knowledge for visual relationships by following the fixed reasoning path of {subject, object <inline-formula> <tex-math notation="LaTeX">\to </tex-math></inline-formula> predicate} to facilitate the recognition of infrequent relationships. However, the knowledge incorporation for such fixed multidependent path suffers from the data set biased and exponentially grown combinations of object and relation labels and ignores the semantic gap between commonsense knowledge and real scenes. To alleviate this, we propose configurable graph reasoning (CGR) to decompose the reasoning path of visual relationships and the incorporation of external knowledge, achieving configurable knowledge selection and personalized graph reasoning for each relation type in each image. Given a commonsense knowledge graph, CGR learns to match and retrieve knowledge for different subpaths and selectively compose the knowledge routed path. CGR adaptively configures the reasoning path based on the knowledge graph, bridges the semantic gap between the commonsense knowledge, and the real-world scenes and achieves better knowledge generalization. Extensive experiments show that CGR consistently outperforms previous state-of-the-art methods on several popular benchmarks and works well with different knowledge graphs. Detailed analyses demonstrated that CGR learned explainable and compelling configurations of reasoning paths.