Deep Modular Bilinear Attention Network for Visual Question Answering

• Published in: Sensors (Basel, Switzerland) Vol. 22; no. 3; p. 1045
• Main Authors: Yan, Feng, Silamu, Wushouer, Li, Yanbing
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 28.01.2022; MDPI
• Subjects: attention mechanism; Bans; bilinear attention network; Language; multi-model; Natural Language Processing; Research Design; visual question answering; bilinear attention network; multi-model; visual question answering; attention mechanism
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s22031045

VQA (Visual Question Answering) is a multi-model task. Given a picture and a question related to the image, it will determine the correct answer. The attention mechanism has become a de facto component of almost all VQA models. Most recent VQA approaches use dot-product to calculate the intra-modality and inter-modality attention between visual and language features. In this paper, the BAN (Bilinear Attention Network) method was used to calculate attention. We propose a deep multimodality bilinear attention network (DMBA-NET) framework with two basic attention units (BAN-GA and BAN-SA) to construct inter-modality and intra-modality relations. The two basic attention units are the core of the whole network framework and can be cascaded in depth. In addition, we encode the question based on the dynamic word vector of BERT(Bidirectional Encoder Representations from Transformers), then use self-attention to process the question features further. Then we sum them with the features obtained by BAN-GA and BAN-SA before the final classification. Without using the Visual Genome datasets for augmentation, the accuracy of our model reaches 70.85% on the test-std dataset of VQA 2.0.