Sarcasm Detection of Dual Multimodal Contrastive Attention Networks

• Published in: 2022 IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications, Digital Twin, Privacy Computing, Metaverse, Autonomous & Trusted Vehicles (SmartWorld/UIC/ScalCom/DigitalTwin/PriComp/Meta) pp. 1455 - 1460
• Main Authors: Lu, Xinkai, Qian, Ying, Yang, Yan, Pang, Wenrao
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2022
• Subjects: deep learning; multimodel; Oral communication; Pricing; sarcasm detection; self-attention; Social networking (online); Vocabulary
• DOI: 10.1109/SmartWorld-UIC-ATC-ScalCom-DigitalTwin-PriComp-Metaverse56740.2022.00210

Sarcasm is a rhetorical method that is commonly used on social media platforms or in daily life to communicate a speaker's feelings of irritation, anger, or mocking, and it often presented in implicit and exaggeration ways. Early sarcasm detection relied on unimodal text identification; however, sarcastic conversations cannot be precisely identified by unimodal model in domains like dialogues system. Most current multimodal studies focus on feature fusion or modality comparison. This method ignored dealing with out-of-vocabulary (OOV) words. For example, this sentence, "All right, Amy's in charge of pricing and being seventy-five." which using the number seventy-five to represent outdated clothing tastes, an obvious sarcasm that would be taken as a neutral word and lead to inconsistencies across different modalities being ignored. We call such words out of vocabulary (OOV) words. In this paper, we propose a dual multimodal contrastive attention network (DMCAN) for sarcasm detection, which consists of two sub-networks containing contrastive attention mechanisms. The first of these networks scores the word polarity of OOV words (including neutral words with sarcasm meaning) and is called the polarity scoring network, while the second network detects sarcasm by inconsistency between different modalities and is called the sarcasm detection network. Our experiments on MUStARD (Multimodal Sarcasm Detection Dataset) prove the effectiveness of the model.