Synthetic Counterfactual Faces

Computer vision systems have been deployed in various applications involving biometrics like human faces. These systems can identify social media users, search for missing persons, and verify identity of individuals. While computer vision models are often evaluated for accuracy on available benchmar...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Ramesh, Guruprasad V, Rosenberg, Harrison, Hooda, Ashish, Shimaa Ahmed Kassem Fawaz
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 29.07.2024
Subjects
Computer vision
Face recognition
Generative artificial intelligence
Image quality
Synthetic data
Vision systems
Online AccessGet full text

Cover

More Information
Summary:Computer vision systems have been deployed in various applications involving biometrics like human faces. These systems can identify social media users, search for missing persons, and verify identity of individuals. While computer vision models are often evaluated for accuracy on available benchmarks, more annotated data is necessary to learn about their robustness and fairness against semantic distributional shifts in input data, especially in face data. Among annotated data, counterfactual examples grant strong explainability characteristics. Because collecting natural face data is prohibitively expensive, we put forth a generative AI-based framework to construct targeted, counterfactual, high-quality synthetic face data. Our synthetic data pipeline has many use cases, including face recognition systems sensitivity evaluations and image understanding system probes. The pipeline is validated with multiple user studies. We showcase the efficacy of our face generation pipeline on a leading commercial vision model. We identify facial attributes that cause vision systems to fail.
ISSN:2331-8422