CA-FER: Mitigating Spurious Correlation With Counterfactual Attention in Facial Expression Recognition

• Published in: IEEE transactions on affective computing Vol. 15; no. 3; pp. 977 - 989
• Main Authors: Huang, Pin-Jui, Xie, Hongxia, Huang, Hung-Cheng, Shuai, Hong-Han, Cheng, Wen-Huang
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.07.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Causal learning; Computational modeling; Correlation; Datasets; Deep learning; Face recognition; facial expression recognition; Feature extraction; Feature recognition; Image color analysis; Reasoning; spurious correlation; Statistical methods; Synthetic data; Training
• ISSN: 1949-3045; 1949-3045
• DOI: 10.1109/TAFFC.2023.3312768

Although facial expression recognition based on deep learning has become a major trend, existing methods have been found to prefer learning spurious statistical correlations and non-robust features during training. This degenerates the model's generalizability in practical situations. One of the research fields mitigating such misperception of correlations as causality is causal reasoning. In this article, we propose a learnable counterfactual attention mechanism, CA-FER, that uses causal reasoning to simultaneously optimize feature discrimination and diversity to mitigate spurious correlations in expression datasets. To the best of our knowledge, this is the first work to study the spurious correlations in facial expression recognition from a counterfactual attention perspective. Extensive experiments on a synthetic dataset and four public datasets demonstrate that our method outperforms previous methods, which shows the effectiveness and generalizability of our learnable counterfactual attention mechanism for the expression recognition task.