Learning to Assemble Neural Module Tree Networks for Visual Grounding

• Published in: Proceedings / IEEE International Conference on Computer Vision pp. 4672 - 4681
• Main Authors: Liu, Daqing, Zhang, Hanwang, Zha, Zheng-Jun, Wu, Feng
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.10.2019
• Subjects: Artificial neural networks; Cognition; Grounding; Natural languages; Task analysis; Training; Visualization
• ISSN: 2380-7504
• DOI: 10.1109/ICCV.2019.00477

Visual grounding, a task to ground (i.e., localize) natural language in images, essentially requires composite visual reasoning. However, existing methods over-simplify the composite nature of language into a monolithic sentence embedding or a coarse composition of subject-predicate-object triplet. In this paper, we propose to ground natural language in an intuitive, explainable, and composite fashion as it should be. In particular, we develop a novel modular network called Neural Module Tree network (NMTree) that regularizes the visual grounding along the dependency parsing tree of the sentence, where each node is a neural module that calculates visual attention according to its linguistic feature, and the grounding score is accumulated in a bottom-up direction where as needed. NMTree disentangles the visual grounding from the composite reasoning, allowing the former to only focus on primitive and easy-to-generalize patterns. To reduce the impact of parsing errors, we train the modules and their assembly end-to-end by using the Gumbel-Softmax approximation and its straight-through gradient estimator, accounting for the discrete nature of module assembly. Overall, the proposed NMTree consistently outperforms the state-of-the-arts on several benchmarks. Qualitative results show explainable grounding score calculation in great detail.