Auditing saliency cropping algorithms

• Published in: Proceedings / IEEE Workshop on Applications of Computer Vision pp. 1515 - 1523
• Main Authors: Birhane, Abeba, Prabhu, Vinay Uday, Whaley, John
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.01.2022
• Subjects: Blogs; Codes; Companies; Computer vision; Explainable AI; Fairness; Accountability; Privacy and Ethics in Vision Datasets; Evaluation and Comparison of Vision Algorithms; Deep Learning; Human-Computer Interaction; Segmentation; Grouping and Shape; Internet; Social networking (online)
• DOI: 10.1109/WACV51458.2022.00158

In this paper, we audit saliency cropping algorithms used by Twitter, Google and Apple to investigate issues pertaining to the male-gaze cropping phenomenon as well as race-gender biases that emerge in post-cropping survival ratios of face-images constituting 3 × 1 grid images. In doing so, we present the first formal empirical study which suggests that the worry of a male-gaze-like image cropping phenomenon on Twitter is not at all far-fetched and it does occur with worryingly high prevalence rates in real-world full-body single-female-subject images shot with logo-littered backdrops. We uncover that while all three saliency cropping frameworks considered in this paper do exhibit acute racial and gender biases, Twitter's saliency cropping framework uniquely elicits high male-gaze cropping prevalence rates. In order to facilitate reproducing the results presented here, we are open-sourcing both the code and the datasets that we curated at shorturl.at/iuzK9. We hope the computer vision community and saliency cropping researchers will build on the results presented here and extend these investigations to similar frameworks deployed in the real world by other companies such as Microsoft and Facebook.