Lensless Sensing of Facial Expression by Transforming Spectral Attention Features

• Published in: IEEE transactions on instrumentation and measurement Vol. 73; pp. 1 - 13
• Main Authors: Yang, Jingyu, Zhang, Mengxi, Yin, Xiangjun, Li, Kun, Yue, Huanjing
• Format: Journal Article
• Language: English
• Published: New York IEEE 2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptive filters; Cameras; Datasets; End-to-end network; Face recognition; facial expression recognition (FER); FlatCam system; Image reconstruction; Imaging; lensless imaging; Modulation; Noise levels; Privacy; Robotics; Task analysis; Transformers
• ISSN: 0018-9456; 1557-9662
• DOI: 10.1109/TIM.2024.3375987

Camera-based facial expression recognition (FER) systems have made significant progress in many areas, such as safe driving and robotics, but they also pose challenges in terms of the imaging system size and privacy-preserving. Emerging lensless cameras are distinguished by their small size and visual privacy due to diffused measurements. Existing lensless FER methods first reconstruct images from lensless measurements and then perform the FER task on reconstructed images. However, these reconstructed images still contain some privacy-sensitive information, which still suffers from privacy leakage. In this article, we propose an end-to-end network called LenslessFET to predict facial expressions directly from lensless measurements without image reconstruction, thus inheriting the privacy-preserving merits of lensless cameras. To this end, we propose the spectral attention (SA) module that learns adaptive filters to extract expression information in the frequency domain. Besides, we observe that SA features contain some undesirable noises that hinder expression recognition. To address the problem of noise interference in SA features, we group them according to their noise level and apply the basis modulation transformer (BMT) to enhance expression information from these noisy features. Extensive experiments show that LenslessFET achieves state-of-the-art (SOTA) performance on the real-captured dataset, that is, FCFD dataset, and simulated FER datasets, that is, RAF-DB<inline-formula> <tex-math notation="LaTeX">^{\dagger} </tex-math></inline-formula> and FERPlus<inline-formula> <tex-math notation="LaTeX">^{\dagger} </tex-math></inline-formula>. Our code will be available at this link.