Multimodal fake news detection through data augmentation-based contrastive learning

• Published in: Applied soft computing Vol. 136; p. 110125
• Main Authors: Hua, Jiaheng, Cui, Xiaodong, Li, Xianghua, Tang, Keke, Zhu, Peican
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.03.2023
• Subjects: Contrastive learning; Fake news detection; Machine learning; Multimodal framework; Fake news detection; Multimodal framework; Machine learning; Contrastive learning
• ISSN: 1568-4946; 1872-9681
• DOI: 10.1016/j.asoc.2023.110125

During the information exploding era, news can be created or edited purposely for promoting the spreading of social influence. However, unverified or fabricated news can also spread unscrupulously, leading to serious consequences, such as poor decisions or even health risk. Thus, in order to discriminate the fake news, several fake news detection approaches have been proposed and the majority of these methods suffer from low efficacy of detection, due to the lack of multimodal information and the small data size. Hence, we develop a novel machine learning based framework, i.e., BERT-based back-Translation Text and Entire-image multimodal model with Contrastive learning (TTEC). In this framework, the text of news is first back-translated encouraging the framework to learn some general characteristics regarding a particular topic. Secondly, both textual and visual features are fed into a BERT-based model in order to produce multimodal features. Thirdly, the contrastive learning is utilized to derive more reasonable multimodal representations through utilizing similar news published in the past. Eventually, to demonstrate the effectiveness of the proposed framework, extensive experiments are conducted and the results show our method outperforms the state of art methods by 3.1% on Mac. F1 scores. •We present a new multimodal fake news detection framework with contrastive learning.•The combination of back-translation and contrastive learning is proved effective.•The specific effects of contrastive learning and different image forms are discussed.•Several SOTA models are compared and extensive experiments are conducted.