Evaluating Generalizability of Fine-Tuned Models for Fake News Detection

The Covid-19 pandemic has caused a dramatic and parallel rise in dangerous misinformation, denoted an `infodemic' by the CDC and WHO. Misinformation tied to the Covid-19 infodemic changes continuously; this can lead to performance degradation of fine-tuned models due to concept drift. Degredati...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Suprem, Abhijit, Calton Pu
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 23.05.2022
Subjects
Cluster analysis
Clustering
Coronaviruses
COVID-19
Datasets
False information
News
Performance degradation
Training
Vector quantization
Online AccessGet full text

Cover

More Information
Summary:The Covid-19 pandemic has caused a dramatic and parallel rise in dangerous misinformation, denoted an `infodemic' by the CDC and WHO. Misinformation tied to the Covid-19 infodemic changes continuously; this can lead to performance degradation of fine-tuned models due to concept drift. Degredation can be mitigated if models generalize well-enough to capture some cyclical aspects of drifted data. In this paper, we explore generalizability of pre-trained and fine-tuned fake news detectors across 9 fake news datasets. We show that existing models often overfit on their training dataset and have poor performance on unseen data. However, on some subsets of unseen data that overlap with training data, models have higher accuracy. Based on this observation, we also present KMeans-Proxy, a fast and effective method based on K-Means clustering for quickly identifying these overlapping subsets of unseen data. KMeans-Proxy improves generalizability on unseen fake news datasets by 0.1-0.2 f1-points across datasets. We present both our generalizability experiments as well as KMeans-Proxy to further research in tackling the fake news problem.
ISSN:2331-8422