SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 11975 - 11985
• Main Authors: Howard, Phillip, Madasu, Avinash, Le, Tiep, Moreno, Gustavo Lujan, Bhiwandiwalla, Anahita, Lal, Vasudev
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Computer vision; counterfactuals; Diffusion models; Fairness; intersectionality; Pattern recognition; Prevention and mitigation; social bias; Text to image; Training; vision-language models

While vision-language models (VLMs) have achieved re-markable performance improvements recently, there is growing evidence that these models also posses harmful biases with respect to social attributes such as gender and race. Prior studies have primarily focused on probing such bias attributes indi...