Assessing bias and computational efficiency in vision transformers using early exits

• Published in: EURASIP journal on image and video processing Vol. 2025; no. 1; pp. 2 - 20
• Main Authors: Nixon, Seth, Ruiu, Pietro, Cadoni, Marinella, Lagorio, Andrea, Tistarelli, Massimo
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 20.01.2025; Springer Nature B.V; SpringerOpen
• Subjects: Bias; Biometrics; Datasets; Deep learning; Early exit; Energy consumption; Engineering; Face recognition; Image Processing and Computer Vision; Machine learning; Matching; Pattern Recognition; Performance evaluation; Reliable and Unbiased Methods for Biometric Recognition; Signal,Image and Speech Processing; Trustworthy; Vision transformer; Biometrics; Vision transformer; Face recognition; Early exit; Bias
• ISSN: 1687-5281; 1687-5176; 1687-5281
• DOI: 10.1186/s13640-024-00658-9

Face recognition with deep learning is generally approached as a problem of capacity. The field has seen progressively deeper, more complex models or larger, more highly variant data sets. The data sets can be problematic, as they are often scraped indiscriminately from the internet. This results in an uncertain, and often heavily unbalanced distribution of race, gender, age and other aspects of the subjects, which is then manifested in the decisions of the models trained on them. The carbon footprint of machine learning is a concern. A real push is developing to reduce the energy consumption of machine learning as we strive for a more eco-friendly society. In addition, due to many instances of misuse by law enforcement and other agencies, unbiased models for face recognition are now fundamental to the practical application of the field. We present an approach using the state of the art Vision Transformer and Early Exits for reducing compute budget without significantly affecting performance. We develop a system for face recognition and identification with a closed-set gallery and show that with a small reduction in performance, a reasonable reduction in compute cost can be obtained using our method. Second, we investigate how these early Exits interact with the bias model through a robust evaluation of matching scores on a racially balanced data set. We show that matching scores vary heavily between cohorts, and these variations are magnified at the earlier exits.