Lensless Sensing of Facial Expression by Transforming Spectral Attention Features
Camera-based facial expression recognition (FER) systems have made significant progress in many areas, such as safe driving and robotics, but they also pose challenges in terms of the imaging system size and privacy-preserving. Emerging lensless cameras are distinguished by their small size and visu...
Saved in:
Published in | IEEE transactions on instrumentation and measurement Vol. 73; pp. 1 - 13 |
---|---|
Main Authors | , , , , |
Format | Journal Article |
Language | English |
Published |
New York
IEEE
2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Camera-based facial expression recognition (FER) systems have made significant progress in many areas, such as safe driving and robotics, but they also pose challenges in terms of the imaging system size and privacy-preserving. Emerging lensless cameras are distinguished by their small size and visual privacy due to diffused measurements. Existing lensless FER methods first reconstruct images from lensless measurements and then perform the FER task on reconstructed images. However, these reconstructed images still contain some privacy-sensitive information, which still suffers from privacy leakage. In this article, we propose an end-to-end network called LenslessFET to predict facial expressions directly from lensless measurements without image reconstruction, thus inheriting the privacy-preserving merits of lensless cameras. To this end, we propose the spectral attention (SA) module that learns adaptive filters to extract expression information in the frequency domain. Besides, we observe that SA features contain some undesirable noises that hinder expression recognition. To address the problem of noise interference in SA features, we group them according to their noise level and apply the basis modulation transformer (BMT) to enhance expression information from these noisy features. Extensive experiments show that LenslessFET achieves state-of-the-art (SOTA) performance on the real-captured dataset, that is, FCFD dataset, and simulated FER datasets, that is, RAF-DB<inline-formula> <tex-math notation="LaTeX">^{\dagger} </tex-math></inline-formula> and FERPlus<inline-formula> <tex-math notation="LaTeX">^{\dagger} </tex-math></inline-formula>. Our code will be available at this link. |
---|---|
AbstractList | Camera-based facial expression recognition (FER) systems have made significant progress in many areas, such as safe driving and robotics, but they also pose challenges in terms of the imaging system size and privacy-preserving. Emerging lensless cameras are distinguished by their small size and visual privacy due to diffused measurements. Existing lensless FER methods first reconstruct images from lensless measurements and then perform the FER task on reconstructed images. However, these reconstructed images still contain some privacy-sensitive information, which still suffers from privacy leakage. In this article, we propose an end-to-end network called LenslessFET to predict facial expressions directly from lensless measurements without image reconstruction, thus inheriting the privacy-preserving merits of lensless cameras. To this end, we propose the spectral attention (SA) module that learns adaptive filters to extract expression information in the frequency domain. Besides, we observe that SA features contain some undesirable noises that hinder expression recognition. To address the problem of noise interference in SA features, we group them according to their noise level and apply the basis modulation transformer (BMT) to enhance expression information from these noisy features. Extensive experiments show that LenslessFET achieves state-of-the-art (SOTA) performance on the real-captured dataset, that is, FCFD dataset, and simulated FER datasets, that is, RAF-DB[Formula Omitted] and FERPlus[Formula Omitted]. Our code will be available at this link. Camera-based facial expression recognition (FER) systems have made significant progress in many areas, such as safe driving and robotics, but they also pose challenges in terms of the imaging system size and privacy-preserving. Emerging lensless cameras are distinguished by their small size and visual privacy due to diffused measurements. Existing lensless FER methods first reconstruct images from lensless measurements and then perform the FER task on reconstructed images. However, these reconstructed images still contain some privacy-sensitive information, which still suffers from privacy leakage. In this article, we propose an end-to-end network called LenslessFET to predict facial expressions directly from lensless measurements without image reconstruction, thus inheriting the privacy-preserving merits of lensless cameras. To this end, we propose the spectral attention (SA) module that learns adaptive filters to extract expression information in the frequency domain. Besides, we observe that SA features contain some undesirable noises that hinder expression recognition. To address the problem of noise interference in SA features, we group them according to their noise level and apply the basis modulation transformer (BMT) to enhance expression information from these noisy features. Extensive experiments show that LenslessFET achieves state-of-the-art (SOTA) performance on the real-captured dataset, that is, FCFD dataset, and simulated FER datasets, that is, RAF-DB<inline-formula> <tex-math notation="LaTeX">^{\dagger} </tex-math></inline-formula> and FERPlus<inline-formula> <tex-math notation="LaTeX">^{\dagger} </tex-math></inline-formula>. Our code will be available at this link. |
Author | Li, Kun Yin, Xiangjun Yue, Huanjing Yang, Jingyu Zhang, Mengxi |
Author_xml | – sequence: 1 givenname: Jingyu orcidid: 0000-0002-7521-7920 surname: Yang fullname: Yang, Jingyu email: yjy@tju.edu.cn organization: School of Electrical and Information Engineering, Tianjin University, Tianjin, China – sequence: 2 givenname: Mengxi orcidid: 0000-0002-6011-1218 surname: Zhang fullname: Zhang, Mengxi email: mengxizhang@tju.edu.cn organization: School of Electrical and Information Engineering, Tianjin University, Tianjin, China – sequence: 3 givenname: Xiangjun orcidid: 0000-0002-4829-9019 surname: Yin fullname: Yin, Xiangjun email: yinxiangjun@tju.edu.cn organization: School of Electrical and Information Engineering, Tianjin University, Tianjin, China – sequence: 4 givenname: Kun orcidid: 0000-0003-2326-0166 surname: Li fullname: Li, Kun email: lik@tju.edu.cn organization: Tianjin Key Laboratory of Cognitive Computing and Application, School of Computer Science and Technology, Tianjin University, Tianjin, China – sequence: 5 givenname: Huanjing orcidid: 0000-0003-2517-9783 surname: Yue fullname: Yue, Huanjing email: huanjing.yue@tju.edu.cn organization: School of Electrical and Information Engineering, Tianjin University, Tianjin, China |
BookMark | eNpNkE1LAzEURYNUsK3uXbgYcD01n5NkWUqrhYpI6zpkkoxMaTNjkoL992ZoF67eg3fufXAmYOQ77wB4RHCGEJQvu_X7DENMZ4RwJgW_AWPEGC9lVeERGEOIRCkpq-7AJMY9hJBXlI_B58b5eHAxFtu8tP676JpipU2rD8Xytw_50na-qM_FLmgfmy4cB2jbO5NCZuYpOZ8GZOV0OmX-Htw2-hDdw3VOwddquVu8lZuP1_VivikNpiyV1jqMkaytIJXQ0jIuiaCY8Nqw2kJdQwmha3hTNVpwJK2wDGnMmMSGMmbIFDxfevvQ_ZxcTGrfnYLPLxWWuUsijkWm4IUyoYsxuEb1oT3qcFYIqkGcyuLUIE5dxeXI0yXSOuf-4ZRTxgX5Ayq1a5U |
CODEN | IEIMAO |
Cites_doi | 10.1609/aaai.v34i07.6930 10.1109/TIM.2022.3204940 10.1007/978-3-642-20465-4_9 10.1109/TCI.2022.3181473 10.1109/CVPR.2017.277 10.1007/978-981-16-5348-3_53 10.1364/OE.27.028075 10.1016/j.ijleo.2012.08.040 10.1109/TIM.2023.3314815 10.1109/TIM.2023.3243661 10.3390/biomimetics8020199 10.1109/TMM.2022.3197365 10.1007/978-3-642-42051-1_16 10.1109/CVPR42600.2020.01400 10.1109/TIFS.2020.3009590 10.1109/CVPR42600.2020.00693 10.1145/3503161.3548303 10.1145/3394171.3413907 10.1109/TCI.2023.3237176 10.1109/TCI.2018.2889933 10.1109/TPAMI.2022.3202765 10.1016/j.patrec.2021.01.029 10.1007/978-3-031-19809-0_24 10.1109/CVPR52688.2022.00413 10.1109/TAFFC.2022.3226473 10.1109/TCI.2016.2593662 10.1038/s41377-022-00809-5 10.1109/TIP.2019.2956143 10.1109/ICASSP49357.2023.10096627 10.1109/TPAMI.2020.2987489 10.1109/WACV48630.2021.00245 10.1109/CVPR42600.2020.00871 10.1109/TCI.2021.3114542 10.1109/TIFS.2019.2946938 10.1145/2993148.2993165 10.1364/OPTICA.5.000001 10.1007/978-0-85729-748-8 10.1109/TAFFC.2017.2740923 10.1109/ICCV.2019.00795 10.1109/TPAMI.2020.3033882 10.1146/annurev-bioeng-092515-010849 10.1109/ICCV48922.2021.00358 |
ContentType | Journal Article |
Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
DBID | 97E RIA RIE AAYXX CITATION 7SP 7U5 8FD L7M |
DOI | 10.1109/TIM.2024.3375987 |
DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005-present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE/IET Electronic Library CrossRef Electronics & Communications Abstracts Solid State and Superconductivity Abstracts Technology Research Database Advanced Technologies Database with Aerospace |
DatabaseTitle | CrossRef Solid State and Superconductivity Abstracts Technology Research Database Advanced Technologies Database with Aerospace Electronics & Communications Abstracts |
DatabaseTitleList | Solid State and Superconductivity Abstracts |
Database_xml | – sequence: 1 dbid: RIE name: IEEE/IET Electronic Library url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ sourceTypes: Publisher |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Engineering Physics |
EISSN | 1557-9662 |
EndPage | 13 |
ExternalDocumentID | 10_1109_TIM_2024_3375987 10474578 |
Genre | orig-research |
GrantInformation_xml | – fundername: National Natural Science Foundation of China grantid: 62231018; 62171317; 62072331 funderid: 10.13039/501100001809 |
GroupedDBID | -~X 0R~ 29I 4.4 5GY 5VS 6IK 85S 8WZ 97E A6W AAJGR AASAJ AAYOK ABQJQ ABVLG ACGFO ACIWK ACNCT AENEX AETIX AI. AIBXA AKJIK ALLEH ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD F5P HZ~ H~9 IAAWW IBMZZ ICLAB IDIHD IFIPE IFJZH IPLJI JAVBF LAI M43 O9- OCL P2P RIA RIE RIG RNS TN5 TWZ VH1 VJK XFK AAYXX CITATION 7SP 7U5 8FD L7M |
ID | FETCH-LOGICAL-c245t-dde2219bd8368a9d579384237bc5bd0ab0900ef7f6fa8719d8d51a25592c455c3 |
IEDL.DBID | RIE |
ISSN | 0018-9456 |
IngestDate | Thu Oct 10 19:29:05 EDT 2024 Fri Aug 23 02:17:01 EDT 2024 Wed Jun 26 19:43:03 EDT 2024 |
IsPeerReviewed | true |
IsScholarly | true |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-c245t-dde2219bd8368a9d579384237bc5bd0ab0900ef7f6fa8719d8d51a25592c455c3 |
ORCID | 0000-0002-6011-1218 0000-0002-4829-9019 0000-0002-7521-7920 0000-0003-2326-0166 0000-0003-2517-9783 |
PQID | 2993891728 |
PQPubID | 85462 |
PageCount | 13 |
ParticipantIDs | ieee_primary_10474578 crossref_primary_10_1109_TIM_2024_3375987 proquest_journals_2993891728 |
PublicationCentury | 2000 |
PublicationDate | 20240000 2024-00-00 20240101 |
PublicationDateYYYYMMDD | 2024-01-01 |
PublicationDate_xml | – year: 2024 text: 20240000 |
PublicationDecade | 2020 |
PublicationPlace | New York |
PublicationPlace_xml | – name: New York |
PublicationTitle | IEEE transactions on instrumentation and measurement |
PublicationTitleAbbrev | TIM |
PublicationYear | 2024 |
Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
References | ref13 ref35 ref12 ref34 ref15 ref37 ref31 ref30 ref11 ref33 ref10 ref32 ref2 ref1 ref17 ref39 ref16 ref38 ref19 ref18 Loshchilov (ref42) 2017 ref24 ref46 ref23 ref45 ref26 ref25 ref20 ref41 ref22 ref44 ref21 ref43 ref28 ref27 ref29 ref7 ref9 ref4 ref3 ref6 ref5 ref40 Mao (ref8) 2023 Bezzam (ref14) 2022 Rao (ref36); 34 |
References_xml | – ident: ref45 doi: 10.1609/aaai.v34i07.6930 – ident: ref27 doi: 10.1109/TIM.2022.3204940 – ident: ref43 doi: 10.1007/978-3-642-20465-4_9 – ident: ref19 doi: 10.1109/TCI.2022.3181473 – ident: ref38 doi: 10.1109/CVPR.2017.277 – ident: ref15 doi: 10.1007/978-981-16-5348-3_53 – ident: ref20 doi: 10.1364/OE.27.028075 – ident: ref21 doi: 10.1016/j.ijleo.2012.08.040 – ident: ref9 doi: 10.1109/TIM.2023.3314815 – ident: ref28 doi: 10.1109/TIM.2023.3243661 – volume: 34 start-page: 980 volume-title: Proc. Adv. Neural Inf. Process. Syst. ident: ref36 article-title: Global filter networks for image classification contributor: fullname: Rao – ident: ref3 doi: 10.3390/biomimetics8020199 – ident: ref6 doi: 10.1109/TMM.2022.3197365 – ident: ref40 doi: 10.1007/978-3-642-42051-1_16 – ident: ref29 doi: 10.1109/CVPR42600.2020.01400 – ident: ref44 doi: 10.1109/TIFS.2020.3009590 – ident: ref2 doi: 10.1109/CVPR42600.2020.00693 – ident: ref46 doi: 10.1145/3503161.3548303 – ident: ref23 doi: 10.1145/3394171.3413907 – ident: ref13 doi: 10.1109/TCI.2023.3237176 – ident: ref17 doi: 10.1109/TCI.2018.2889933 – ident: ref41 doi: 10.1109/TPAMI.2022.3202765 – ident: ref4 doi: 10.1016/j.patrec.2021.01.029 – ident: ref5 doi: 10.1007/978-3-031-19809-0_24 – year: 2017 ident: ref42 article-title: Decoupled weight decay regularization publication-title: arXiv:1711.05101 contributor: fullname: Loshchilov – ident: ref30 doi: 10.1109/CVPR52688.2022.00413 – ident: ref7 doi: 10.1109/TAFFC.2022.3226473 – ident: ref10 doi: 10.1109/TCI.2016.2593662 – ident: ref34 doi: 10.1038/s41377-022-00809-5 – ident: ref1 doi: 10.1109/TIP.2019.2956143 – ident: ref16 doi: 10.1109/ICASSP49357.2023.10096627 – year: 2022 ident: ref14 article-title: Learning rich optical embeddings for privacy-preserving lensless image classification publication-title: arXiv:2206.01429 contributor: fullname: Bezzam – ident: ref11 doi: 10.1109/TPAMI.2020.2987489 – ident: ref24 doi: 10.1109/WACV48630.2021.00245 – ident: ref35 doi: 10.1109/CVPR42600.2020.00871 – year: 2023 ident: ref8 article-title: POSTER++: A simpler and stronger facial expression recognition network publication-title: arXiv:2301.12149 contributor: fullname: Mao – ident: ref32 doi: 10.1109/TCI.2021.3114542 – ident: ref37 doi: 10.1109/TIFS.2019.2946938 – ident: ref39 doi: 10.1145/2993148.2993165 – ident: ref12 doi: 10.1364/OPTICA.5.000001 – ident: ref22 doi: 10.1007/978-0-85729-748-8 – ident: ref25 doi: 10.1109/TAFFC.2017.2740923 – ident: ref33 doi: 10.1109/ICCV.2019.00795 – ident: ref18 doi: 10.1109/TPAMI.2020.3033882 – ident: ref31 doi: 10.1146/annurev-bioeng-092515-010849 – ident: ref26 doi: 10.1109/ICCV48922.2021.00358 |
SSID | ssj0007647 |
Score | 2.443744 |
Snippet | Camera-based facial expression recognition (FER) systems have made significant progress in many areas, such as safe driving and robotics, but they also pose... |
SourceID | proquest crossref ieee |
SourceType | Aggregation Database Publisher |
StartPage | 1 |
SubjectTerms | Adaptive filters Cameras Datasets End-to-end network Face recognition facial expression recognition (FER) FlatCam system Image reconstruction Imaging lensless imaging Modulation Noise levels Privacy Robotics Task analysis Transformers |
Title | Lensless Sensing of Facial Expression by Transforming Spectral Attention Features |
URI | https://ieeexplore.ieee.org/document/10474578 https://www.proquest.com/docview/2993891728 |
Volume | 73 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LS8NAEB60IOjBZ8VqlT148ZB2s8kmm2ORliq2IFboLewrHpRUbArqr3cnDymK4C2QTVh2Zmfn8c23AJcmZLHPVODFEZJqZ1J7yqfayyKmeKJiaS1WdCfTaPwY3s75vG5WL3thrLUl-Mz28LGs5ZuFXmGqrI-0AqFTsU3YFJRVzVrfZjeOwoog03c72LkFTU2SJv3ZzcRFgizsBUHME0TPrZ1B5aUqvyxxebyM9mDaTKxClTz3VoXq6c8fnI3_nvk-7NaOJhlUmnEAGzY_hJ01-sFD2Crhn3p5BPd3Lpp9cUaPPCCgPX8ii4yMJKbTyfC9xsrmRH2QWePo4iC8vB4zJWRQFBVskqBLuXLj2_A4Gs6ux1592YKnWcgLz5k55qyXMiKIhEwMdxtXIGZGaa4MlYomlNoszqJMuiArMcJwX2JAwnTIuQ6OoZUvcnsChAuhjQ4CH8M1LoUKuOLGSE0z9yYzHbhqlj99rTg10jIWoUnqRJWiqNJaVB1o42qujasWsgPdRmBpveuWqTtasewaM3H6x2dnsI1_r3IoXWgVbyt77ryKQl2U2vQFQ4jJNA |
link.rule.ids | 315,786,790,802,4043,27954,27955,27956,55107 |
linkProvider | IEEE |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LS8NAEB60IurBR61YrboHLx7Sbh6bx7FIS6ttQWyht7CveFBSsSmov96dPKQogrdAJmTZ2Z3nNzMA18pzAtsRrhX42FQ74dISNpVW4juCRSLgWmNGdzzxBzPvbs7mZbF6Xgujtc7BZ7qNj3kuXy3kCkNlHWwr4JkjtglbRtHToCjX-ha8ge8VLTJtc4eNYVBlJWnUmQ7Hxhd0vLbrBixC_NyaFsrHqvySxbmC6R_ApFpagSt5bq8y0ZafP7o2_nvth7BfmpqkW5yNI9jQaR321hoQ1mE7B4DK5TE8jIw_-2LEHnlESHv6RBYJ6XMMqJPee4mWTYn4INPK1EUiHF-PsRLSzbICOEnQqFwZ-gbM-r3p7cAqxy1Y0vFYZhlB5xj5JVTo-iGPFDNXN0TUjJBMKMoFjSjVSZD4CTduVqRCxWyOLokjPcakewK1dJHqUyAsDKWSrmujw8Z4KFwmmFJc0sS8SVQTbqrtj1-Lrhpx7o3QKDasipFVccmqJjRwN9foio1sQqtiWFzeu2VslCsmXgMnPPvjsyvYGUzHo3g0nNyfwy7-qYiotKCWva30hbExMnGZn6wvCgrMiA |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Lensless+Sensing+of+Facial+Expression+by+Transforming+Spectral+Attention+Features&rft.jtitle=IEEE+transactions+on+instrumentation+and+measurement&rft.au=Yang%2C+Jingyu&rft.au=Zhang%2C+Mengxi&rft.au=Yin%2C+Xiangjun&rft.au=Li%2C+Kun&rft.date=2024&rft.issn=0018-9456&rft.eissn=1557-9662&rft.volume=73&rft.spage=1&rft.epage=13&rft_id=info:doi/10.1109%2FTIM.2024.3375987&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TIM_2024_3375987 |
thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9456&client=summon |
thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9456&client=summon |
thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9456&client=summon |