Wearable Eye-Tracking System for Synchronized Multimodal Data Acquisition

Eye-tracking technology is extensively utilized in affective computing research, enabling the investigation of emotional responses through the analysis of eye movements. Integration of eye-tracking with other modalities, allows for the collection of multimodal data, leading to a more comprehensive u...

Full description

Saved in:
Bibliographic Details
Published inIEEE transactions on circuits and systems for video technology Vol. 34; no. 6; pp. 5146 - 5159
Main Authors Yang, Minqiang, Gao, Yujie, Tang, Longzhe, Hou, Jian, Hu, Bin
Format Journal Article
LanguageEnglish
Published New York IEEE 01.06.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects
Affective computing
Cameras
Cost analysis
Data acquisition
Data collection
Emotional factors
Eye movements
Gaze tracking
hard real-time synchronization
Occlusion
Physiological responses
Solid modeling
Synchronization
Three dimensional models
Three-dimensional displays
Time synchronization
Tracking
Tracking systems
Wearable devices
Wearable eye tracker
Online AccessGet full text

Cover

Abstract Eye-tracking technology is extensively utilized in affective computing research, enabling the investigation of emotional responses through the analysis of eye movements. Integration of eye-tracking with other modalities, allows for the collection of multimodal data, leading to a more comprehensive understanding of emotions and their relationship with physiological responses. This paper presents a novel head-mounted eye-tracking system for multimodal data acquisition with a completely redesigned structure and improved performance. We propose a novel method for pupil-fitting with high efficiency and robustness based on deep learning and RANSAC, which gets better performance of pupil segmentation when it is partially occluded, and build a 3D model to obtain gaze points. Existing eye trackers for multi-modal synchronous data collection either have limited device support or suffer from significant synchronization delays. Our proposed hard real-time synchronization mechanism implements microsecond level latency with low cost, which facilitates multimodal analysis for affective computing research. The uniquely designed exterior effectively reduces facial occlusion, making it more comfortable for the wearer while facilitating the capture of facial expressions.
AbstractList Eye-tracking technology is extensively utilized in affective computing research, enabling the investigation of emotional responses through the analysis of eye movements. Integration of eye-tracking with other modalities, allows for the collection of multimodal data, leading to a more comprehensive understanding of emotions and their relationship with physiological responses. This paper presents a novel head-mounted eye-tracking system for multimodal data acquisition with a completely redesigned structure and improved performance. We propose a novel method for pupil-fitting with high efficiency and robustness based on deep learning and RANSAC, which gets better performance of pupil segmentation when it is partially occluded, and build a 3D model to obtain gaze points. Existing eye trackers for multi-modal synchronous data collection either have limited device support or suffer from significant synchronization delays. Our proposed hard real-time synchronization mechanism implements microsecond level latency with low cost, which facilitates multimodal analysis for affective computing research. The uniquely designed exterior effectively reduces facial occlusion, making it more comfortable for the wearer while facilitating the capture of facial expressions.
Author Gao, Yujie
Hou, Jian
Yang, Minqiang
Tang, Longzhe
Hu, Bin
Author_xml – sequence: 1
  givenname: Minqiang
  orcidid: 0000-0002-7571-6439
  surname: Yang
  fullname: Yang, Minqiang
  email: yangmq@lzu.edu.cn
  organization: School of information Science and Engineering, Lanzhou University, Lanzhou, China
– sequence: 2
  givenname: Yujie
  orcidid: 0009-0002-6256-0401
  surname: Gao
  fullname: Gao, Yujie
  organization: School of information Science and Engineering, Lanzhou University, Lanzhou, China
– sequence: 3
  givenname: Longzhe
  surname: Tang
  fullname: Tang, Longzhe
  organization: School of information Science and Engineering, Lanzhou University, Lanzhou, China
– sequence: 4
  givenname: Jian
  surname: Hou
  fullname: Hou, Jian
  organization: School of information Science and Engineering, Lanzhou University, Lanzhou, China
– sequence: 5
  givenname: Bin
  orcidid: 0000-0003-3514-5413
  surname: Hu
  fullname: Hu, Bin
  email: bh@lzu.edu.cn
  organization: School of information Science and Engineering, Lanzhou University, Lanzhou, China
BookMark eNp9kLtOwzAUhi1UJErhBRBDJOYUX-I4GatSoFIRQwuMluucgEsat7YzlKfHpR0QA9P5h_Ody3eOeq1tAaErgoeE4PJ2MZ6_LoYUUzZkjNGCZCeoTzgvUkox78WMOUkLSvgZOvd-hTHJikz00fQNlFPLBpLJDtKFU_rTtO_JfOcDrJPauhhb_eFsa76gSp66Jpi1rVST3KmgkpHedsabYGx7gU5r1Xi4PNYBermfLMaP6ez5YToezVJNyzykeY41zgWBmmC2zGPOqrJUFS-VhhpKUQutMs4qCgQLxZc8q4AIUWa1JpQBG6Cbw9yNs9sOfJAr27k2rpQsDhPx66hggIpDl3bWewe11Cao_Z3BKdNIguVenPwRJ_fi5FFcROkfdOPMWrnd_9D1ATIA8AtgpMBlzr4BqMJ72Q
CODEN ITCTEM
CitedBy_id crossref_primary_10_1109_TCSS_2024_3405949
crossref_primary_10_3758_s13428_024_02529_7
crossref_primary_10_1109_JSEN_2024_3492007
crossref_primary_10_3389_fpsyg_2024_1253045
crossref_primary_10_1016_j_bspc_2025_107669
crossref_primary_10_1109_JBHI_2024_3428512
crossref_primary_10_1109_TCSS_2024_3420445
crossref_primary_10_1109_TIM_2025_3542109
crossref_primary_10_1109_TCSS_2024_3397403
crossref_primary_10_3390_mti9010006
Cites_doi 10.1109/ICCE-TW.2016.7520987
10.1145/2578153.2578202
10.1109/TBME.2007.895750
10.1109/AFGR.2000.840622
10.1109/TCSVT.2014.2329362
10.1109/CVPR.2004.366
10.1016/j.patcog.2009.08.016
10.3758/s13428-021-01657-8
10.1016/j.jneumeth.2019.05.016
10.1145/2638728.2641695
10.1109/TCSVT.2023.3258962
10.1145/3297714
10.3390/electronics10111331
10.1038/s41598-023-38346-9
10.1109/TCSVT.2022.3152800
10.1145/2857491.2857530
10.1109/TSMCB.2003.811128
10.16910/jemr.7.1.4
10.1109/ACV.2002.1182180
10.1007/s11831-021-09587-6
10.1109/TCSVT.2016.2595322
10.1109/TCSVT.2023.3239381
10.1016/j.chb.2023.107720
10.1038/s41598-018-23265-x
10.1109/TII.2019.2933481
10.1109/TIP.2018.2851672
10.1109/TCSVT.2011.2133770
10.1109/IROS40897.2019.8968536
10.1109/TCSVT.2018.2844780
10.1038/369742a0
10.1145/2168556.2168585
10.1109/ETFA.2018.8502456
10.3390/s18092826
10.1109/RTCSA.1998.726346
10.1145/3314111.3319819
10.4324/9780203488430
10.1109/TPAMI.2009.30
10.3390/brainsci10080558
10.1080/08990220.2018.1547697
10.3390/s20082384
10.1145/358669.358692
10.1109/TBME.2005.863952
10.3390/app6060174
10.1109/ICPR.2000.902895
10.1109/CVPR.2003.1211502
10.3758/s13428-022-02010-3
10.1109/ACCESS.2017.2735633
10.1109/TCSVT.2021.3061719
10.12700/APH.17.2.2020.2.4
10.7717/peerj.9414
10.1109/TCSS.2023.3268505
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
DBID 97E
RIA
RIE
AAYXX
CITATION
7SC
7SP
8FD
JQ2
L7M
L~C
L~D
DOI 10.1109/TCSVT.2023.3332814
DatabaseName IEEE All-Society Periodicals Package (ASPP) 2005-present
IEEE All-Society Periodicals Package (ASPP) 1998-Present
IEEE Electronic Library (IEL)
CrossRef
Computer and Information Systems Abstracts
Electronics & Communications Abstracts
Technology Research Database
ProQuest Computer Science Collection
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts – Academic
Computer and Information Systems Abstracts Professional
DatabaseTitle CrossRef
Technology Research Database
Computer and Information Systems Abstracts – Academic
Electronics & Communications Abstracts
ProQuest Computer Science Collection
Computer and Information Systems Abstracts
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts Professional
DatabaseTitleList
Technology Research Database
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Xplore
  url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 1558-2205
EndPage 5159
ExternalDocumentID 10_1109_TCSVT_2023_3332814
10318096
Genre orig-research
GrantInformation_xml – fundername: Fundamental Research Funds for the Central Universities
  grantid: lzujbky-2023-16
  funderid: 10.13039/501100012226
– fundername: National Key Research and Development Program of China
  grantid: 2019YFA0706200
  funderid: 10.13039/501100012166
– fundername: Supercomputing Center, Lanzhou University
– fundername: Natural Science Foundation of Gansu Province; Natural Science Foundation of Gansu Province, China
  grantid: 22JR5RA488
  funderid: 10.13039/501100004775
– fundername: National Natural Science Foundation of China
  grantid: 62227807
  funderid: 10.13039/501100001809
GroupedDBID -~X
0R~
29I
4.4
5GY
5VS
6IK
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACGFO
ACGFS
ACIWK
AENEX
AETIX
AGQYO
AGSQL
AHBIQ
AI.
AIBXA
AKJIK
AKQYR
ALLEH
ALMA_UNASSIGNED_HOLDINGS
ASUFR
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
DU5
EBS
EJD
HZ~
H~9
ICLAB
IFIPE
IFJZH
IPLJI
JAVBF
LAI
M43
O9-
OCL
P2P
RIA
RIE
RNS
RXW
TAE
TN5
VH1
AAYXX
CITATION
RIG
7SC
7SP
8FD
JQ2
L7M
L~C
L~D
ID FETCH-LOGICAL-c296t-660c0671ef103b6c064d99ad59acefe97f7ca453d2e107a5b54de17794fc123e3
IEDL.DBID RIE
ISSN 1051-8215
IngestDate Sun Jun 29 16:43:05 EDT 2025
Thu Apr 24 22:55:38 EDT 2025
Tue Jul 01 00:41:24 EDT 2025
Wed Aug 27 01:58:40 EDT 2025
IsPeerReviewed true
IsScholarly true
Issue 6
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
https://doi.org/10.15223/policy-029
https://doi.org/10.15223/policy-037
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c296t-660c0671ef103b6c064d99ad59acefe97f7ca453d2e107a5b54de17794fc123e3
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0003-3514-5413
0000-0002-7571-6439
0009-0002-6256-0401
PQID 3064715533
PQPubID 85433
PageCount 14
ParticipantIDs crossref_primary_10_1109_TCSVT_2023_3332814
proquest_journals_3064715533
ieee_primary_10318096
crossref_citationtrail_10_1109_TCSVT_2023_3332814
ProviderPackageCode CITATION
AAYXX
PublicationCentury 2000
PublicationDate 2024-06-01
PublicationDateYYYYMMDD 2024-06-01
PublicationDate_xml – month: 06
  year: 2024
  text: 2024-06-01
  day: 01
PublicationDecade 2020
PublicationPlace New York
PublicationPlace_xml – name: New York
PublicationTitle IEEE transactions on circuits and systems for video technology
PublicationTitleAbbrev TCSVT
PublicationYear 2024
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References ref13
ref57
ref12
ref56
ref15
ref59
ref14
ref58
ref53
ref52
ref11
ref55
ref10
ref54
ref17
ref16
ref19
(ref47) 2023
(ref37) 2020
ref51
Delvigne (ref32) 2023
ref50
ref46
ref45
ref48
ref42
ref44
Yanoff (ref6) 2008
ref49
ref8
ref7
ref9
ref4
ref3
ref5
Senior (ref41) 2010
ref40
Swirski (ref18)
ref35
ref34
ref36
ref31
ref30
ref33
ref2
ref1
Schneider (ref43) 2012
Le Grand (ref39) 1957
ref24
ref23
ref26
ref25
ref20
ref22
Derpanis (ref38) 2010; 4
ref21
ref28
ref27
ref29
Reinders (ref60)
ref61
References_xml – ident: ref23
  doi: 10.1109/ICCE-TW.2016.7520987
– ident: ref56
  doi: 10.1145/2578153.2578202
– ident: ref42
  doi: 10.1109/TBME.2007.895750
– ident: ref55
  doi: 10.1109/AFGR.2000.840622
– ident: ref59
  doi: 10.1109/TCSVT.2014.2329362
– ident: ref50
  doi: 10.1109/CVPR.2004.366
– ident: ref48
  doi: 10.1016/j.patcog.2009.08.016
– ident: ref30
  doi: 10.3758/s13428-021-01657-8
– ident: ref49
  doi: 10.1016/j.jneumeth.2019.05.016
– ident: ref61
  doi: 10.1145/2638728.2641695
– ident: ref26
  doi: 10.1109/TCSVT.2023.3258962
– ident: ref34
  doi: 10.1145/3297714
– start-page: 111
  volume-title: Proc. PETMEI
  ident: ref18
  article-title: A fully-automatic, temporal approach to single camera, glint-free 3D eye model fitting
– ident: ref46
  doi: 10.3390/electronics10111331
– ident: ref5
  doi: 10.1038/s41598-023-38346-9
– ident: ref27
  doi: 10.1109/TCSVT.2022.3152800
– ident: ref22
  doi: 10.1145/2857491.2857530
– ident: ref53
  doi: 10.1109/TSMCB.2003.811128
– ident: ref51
  doi: 10.16910/jemr.7.1.4
– ident: ref58
  doi: 10.1109/ACV.2002.1182180
– ident: ref36
  doi: 10.1007/s11831-021-09587-6
– ident: ref57
  doi: 10.1109/TCSVT.2016.2595322
– ident: ref15
  doi: 10.1109/TCSVT.2023.3239381
– volume-title: The Eye: The Physiology of Human Perception
  year: 2010
  ident: ref41
– ident: ref9
  doi: 10.1016/j.chb.2023.107720
– ident: ref14
  doi: 10.1038/s41598-018-23265-x
– ident: ref20
  doi: 10.1109/TII.2019.2933481
– ident: ref25
  doi: 10.1109/TIP.2018.2851672
– ident: ref11
  doi: 10.1109/TCSVT.2011.2133770
– ident: ref19
  doi: 10.1109/IROS40897.2019.8968536
– volume-title: Ultralytics, YOLOv5, GitHub Repository
  year: 2020
  ident: ref37
– ident: ref12
  doi: 10.1109/TCSVT.2018.2844780
– ident: ref28
  doi: 10.1038/369742a0
– ident: ref17
  doi: 10.1145/2168556.2168585
– volume: 4
  start-page: 23
  issue: 1
  year: 2010
  ident: ref38
  article-title: Overview of the RANSAC algorithm
  publication-title: Image Rochester NY
– ident: ref45
  doi: 10.1109/ETFA.2018.8502456
– ident: ref4
  doi: 10.3390/s18092826
– ident: ref35
  doi: 10.1109/RTCSA.1998.726346
– ident: ref29
  doi: 10.1145/3314111.3319819
– volume-title: Light, Colour and Vision
  year: 1957
  ident: ref39
– ident: ref1
  doi: 10.4324/9780203488430
– ident: ref16
  doi: 10.1109/TPAMI.2009.30
– ident: ref2
  doi: 10.3390/brainsci10080558
– ident: ref8
  doi: 10.1080/08990220.2018.1547697
– ident: ref13
  doi: 10.3390/s20082384
– ident: ref33
  doi: 10.1145/358669.358692
– ident: ref40
  doi: 10.1109/TBME.2005.863952
– volume-title: CASIA.v4
  year: 2023
  ident: ref47
– ident: ref24
  doi: 10.3390/app6060174
– ident: ref54
  doi: 10.1109/ICPR.2000.902895
– start-page: 85
  volume-title: Proc. 3rd Annu. Conf. ASCI, ASCI, Delft
  ident: ref60
  article-title: Eye tracking by template matching using an automatic codebook generation scheme
– year: 2023
  ident: ref32
  article-title: Evaluating the Tobii Pro Glasses 2 and 3 in static and dynamic conditions
  publication-title: Behav. Res. Methods
– ident: ref52
  doi: 10.1109/CVPR.2003.1211502
– ident: ref31
  doi: 10.3758/s13428-022-02010-3
– volume-title: E-Prime Users Guide
  year: 2012
  ident: ref43
– ident: ref21
  doi: 10.1109/ACCESS.2017.2735633
– ident: ref7
  doi: 10.1109/TCSVT.2021.3061719
– ident: ref10
  doi: 10.12700/APH.17.2.2020.2.4
– volume-title: Ophthalmology
  year: 2008
  ident: ref6
– ident: ref44
  doi: 10.7717/peerj.9414
– ident: ref3
  doi: 10.1109/TCSS.2023.3268505
SSID ssj0014847
Score 2.563425
Snippet Eye-tracking technology is extensively utilized in affective computing research, enabling the investigation of emotional responses through the analysis of eye...
SourceID proquest
crossref
ieee
SourceType Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 5146
SubjectTerms Affective computing
Cameras
Cost analysis
Data acquisition
Data collection
Emotional factors
Eye movements
Gaze tracking
hard real-time synchronization
Occlusion
Physiological responses
Solid modeling
Synchronization
Three dimensional models
Three-dimensional displays
Time synchronization
Tracking
Tracking systems
Wearable devices
Wearable eye tracker
Title Wearable Eye-Tracking System for Synchronized Multimodal Data Acquisition
URI https://ieeexplore.ieee.org/document/10318096
https://www.proquest.com/docview/3064715533
Volume 34
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV07T8MwELZoJxh4IwoFZWBDDk3sPDwiaAVIsFAeW-THWUJAyyMZ6K_n7KRVAYHYPDiW5bPz3dn33UfIgUqNlRBpKpmKKdcmpSLRljIZ58qYLJPS8Z0vr9KzG35xn9w3ZHXPhQEAn3wGoWv6t3wz1pW7KjtykgQ5-twt0sLIrSZrzZ4MeO7VxNBfiGiOQDZlyPTE0fDk-nYYOqHwkDEW5xH_gkJeVuXHv9gDzGCFXE2nVueVPIZVqUI9-Va18d9zXyXLjasZHNd7Y40swGidLM0VINwg53e40R15Kuh_AEXc0u7mPKjLmAfoz2JzpH393AmYwLN1n8cGRz2VpQyO9Wv1UCd9bZKbQX94ckYbcQWqY5GWNE17GpEqAoszUym2uRFCmkRIDRZEZjMtecJMDBghykQl3ECU4fG1GtEO2BZpj8Yj2CZBzgwGRj0Q1iAmZjy3qQGrjEhElCsFHRJNF7vQTeVxJ4DxVPgIpCcKb6DCGahoDNQhh7NvXuq6G3_23nQrPtezXuwO6U6NWjRn871wMVfm5JLYzi-f7ZJFHJ3XGWFd0i7fKthD36NU-37PfQI8BtXE
linkProvider IEEE
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LT9wwEB7xOAAHWh4V225LDr0hh03sPHxEW9Dy2kuXxy3yYyyhwi6P7KH76xk7WUSpqHrzwUksj51vxp5vPoDvOrdOYWKY4jplwticycw4xlVaamuLQinPdz4f5oMLcXKdXbdk9cCFQcSQfIaxb4a7fDsxU39Utu8lCUryuRdhmYA_Sxq61sulgSiDnhh5DAkrCcrmHJme3B_1f16OYi8VHnPO0zIRf-BQEFb5628cIOboAwzng2syS37F01rHZvambuN_j_4jrLfOZnTQrI4NWMDxJqy9KkG4BcdXtNQ9fSo6_I2MkMv4s_OoKWQekUdLzbEJFXRnaKPA172bWHrrD1Wr6MA8TG-atK9tuDg6HPUHrJVXYCaVec3yvGcIqxJ0NDKdU1tYKZXNpDLoUBauMEpk3KZIMaLKdCYsJgVtYGcI75B_gqXxZIw7EJXcUmjUQ-ksoWIhSpdbdNrKTCal1tiBZD7ZlWlrj3sJjNsqxCA9WQUDVd5AVWugDuy9PHPfVN74Z-9tP-OvejaT3YHu3KhVuzufKh91FV4wiX9-57FdWBmMzs-qs-Ph6RdYpS-JJj-sC0v14xS_kidS629h_T0Djx3ZDQ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Wearable+Eye-Tracking+System+for+Synchronized+Multimodal+Data+Acquisition&rft.jtitle=IEEE+transactions+on+circuits+and+systems+for+video+technology&rft.au=Yang%2C+Minqiang&rft.au=Gao%2C+Yujie&rft.au=Tang%2C+Longzhe&rft.au=Hou%2C+Jian&rft.date=2024-06-01&rft.pub=IEEE&rft.issn=1051-8215&rft.volume=34&rft.issue=6&rft.spage=5146&rft.epage=5159&rft_id=info:doi/10.1109%2FTCSVT.2023.3332814&rft.externalDocID=10318096
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1051-8215&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1051-8215&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1051-8215&client=summon