High-Order Interaction Learning for Image Captioning

• Published in: IEEE transactions on circuits and systems for video technology Vol. 32; no. 7; pp. 4417 - 4430
• Main Authors: Wang, Yanhui, Xu, Ning, Liu, An-An, Li, Wenhui, Zhang, Yongdong
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.07.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Ablation; Coders; Context; Cutlery; Cutting; Decoding; encoder-decoder framework; Encoders-Decoders; Encoding; Feature extraction; high-order interaction; Image captioning; Ions; Learning; Object recognition; Semantics; Sentences; Task analysis; Triplets; Visualization
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2021.3121062

Image captioning aims at understanding various semantic concepts (e.g., objects and relationships) from an image and integrating them in a sentence-level description. Hence, it is necessary to learn the interaction among these concepts. If we define the context of the interaction to be involved in the subject-predicate-object triplet, most current methods only focus on the single triplet for the first-order interaction to generate sentences. Intuitively, we humans are able to perceive the high-order interaction among concepts from two or more triplets to describe an image. For example, when we see the triplets man-cutting-sandwich and man-with-knife , it is natural to integrate and predict the sentence man cutting sandwich with knife . This depends on the high-order interaction between cutting and knife in different triplets. Therefore, exploiting high-order interaction is expected to benefit image captioning and focus on reasoning. In this paper, we introduce the novel high-order interaction learning method over detected objects and relationships for image captioning under the umbrella of the encoder-decoder framework. We first extract a set of object and relationship features in an image. During the encoding stage, the interactive refining network is proposed to learn high-order representations by modeling intra- and inter-object feature interaction in the self-attention fashion. During the decoding stage, the interactive fusion network is proposed to integrate object and relationship information by strengthening their high-order interaction based on language context for sentence generation. In this way, we learn the object-relationship dependencies in different stages, which can provide abundant cues for both visual understanding and caption generation. Extensive experiments show that the proposed method can achieve competitive performances against the state-of-the-art methods on MSCOCO dataset. Additional ablation studies further validate its effectiveness.