Enhancing the Applicability of Sign Language Translation

This paper addresses a significant problem in American Sign Language (ASL) translation systems that has been overlooked. Current designs collect excessive sensing data for each word and treat every sentence as new, requiring the collection of sensing data from scratch. This approach is time-consumin...

Full description

Saved in:
Bibliographic Details
Published inIEEE transactions on mobile computing Vol. 23; no. 9; pp. 8634 - 8648
Main Authors Li, Jiao, Xu, Jiakai, Liu, Yang, Xu, Weitao, Li, Zhenjiang
Format Magazine Article
LanguageEnglish
Published IEEE 01.09.2024
Subjects
Assistive technologies
Computer science
Gesture recognition
Libraries
Mobile computing
Semantics
Sensors
sign language translation
Urban areas
wearable sensing
Online AccessGet full text

Cover

Abstract This paper addresses a significant problem in American Sign Language (ASL) translation systems that has been overlooked. Current designs collect excessive sensing data for each word and treat every sentence as new, requiring the collection of sensing data from scratch. This approach is time-consuming, taking hours to half a day to complete the data collection process for each user. As a result, it creates an unnecessary burden on end-users and hinders the widespread adoption of ASL systems. In this study, we identify the root cause of this issue and propose GASLA -a wearable sensor-based solution that automatically generates sentence-level sensing data from word-level data. An acceleration approach is further proposed to optimize the data generation speed. Moreover, due to the gap between the generated sentence data and directly collected sentence data, a template strategy is proposed to make the generated sentences more similar to the collected sentence. The generated data can be used to train ASL systems effectively while reducing overhead costs significantly. GASLA offers several benefits over current approaches: it reduces initial setup time and future new-sentence addition overhead; it requires only two samples per sentence compared to around ten samples in current systems; and it improves overall performance significantly.
AbstractList This paper addresses a significant problem in American Sign Language (ASL) translation systems that has been overlooked. Current designs collect excessive sensing data for each word and treat every sentence as new, requiring the collection of sensing data from scratch. This approach is time-consuming, taking hours to half a day to complete the data collection process for each user. As a result, it creates an unnecessary burden on end-users and hinders the widespread adoption of ASL systems. In this study, we identify the root cause of this issue and propose GASLA -a wearable sensor-based solution that automatically generates sentence-level sensing data from word-level data. An acceleration approach is further proposed to optimize the data generation speed. Moreover, due to the gap between the generated sentence data and directly collected sentence data, a template strategy is proposed to make the generated sentences more similar to the collected sentence. The generated data can be used to train ASL systems effectively while reducing overhead costs significantly. GASLA offers several benefits over current approaches: it reduces initial setup time and future new-sentence addition overhead; it requires only two samples per sentence compared to around ten samples in current systems; and it improves overall performance significantly.
Author Li, Zhenjiang
Liu, Yang
Li, Jiao
Xu, Jiakai
Xu, Weitao
Author_xml – sequence: 1
  givenname: Jiao
  orcidid: 0000-0002-3918-4922
  surname: Li
  fullname: Li, Jiao
  email: jli242-c@my.cityu.edu.hk
  organization: Department of Computer Science, City University of Hong Kong, Hong Kong, SAR, China
– sequence: 2
  givenname: Jiakai
  orcidid: 0009-0006-2074-9109
  surname: Xu
  fullname: Xu, Jiakai
  email: jiakai.xu@my.cityu.edu.hk
  organization: Department of Computer Science, City University of Hong Kong, Hong Kong, SAR, China
– sequence: 3
  givenname: Yang
  orcidid: 0000-0002-2474-2004
  surname: Liu
  fullname: Liu, Yang
  email: yl868@cam.ac.uk
  organization: Department of Computer Science and Technology, University of Cambridge, Cambridge, England
– sequence: 4
  givenname: Weitao
  orcidid: 0000-0001-9741-5912
  surname: Xu
  fullname: Xu, Weitao
  email: weitaoxu@cityu.edu.hk
  organization: Department of Computer Science, City University of Hong Kong, Hong Kong, SAR, China
– sequence: 5
  givenname: Zhenjiang
  orcidid: 0000-0002-3296-3392
  surname: Li
  fullname: Li, Zhenjiang
  email: zhenjiang.li@cityu.edu.hk
  organization: Department of Computer Science, City University of Hong Kong, Hong Kong, SAR, China
BookMark eNpNjzFPwzAQhS1UJNrCzsCQP5ByZzuxM1ZRoUhBDIQ5chw7NQpOFIeh_55E7cD0Tk_v3d23ISvfe0PII8IOEbLn8j3fUaB8x1gCiHhD1pgkMoY0hdUyszRGytgd2YTwDYAyy8SayIM_Ka-db6PpZKL9MHROq9p1bjpHvY0-XeujQvn2V7UmKkflQ6cm1_t7cmtVF8zDVbfk6-VQ5se4-Hh9y_dFrCmKKdaWWZ5ibcFImyRCCyEakVikWimkXHOhsebKCJASpeSsbmbfIFpo6vnfLYHLXj32IYzGVsPoftR4rhCqhbyayauFvLqSz5WnS8UZY_7F2XwAGPsDxG5WAQ
CODEN ITMCCJ
Cites_doi 10.1145/3472291
10.1109/INFOCOM.2017.8057209
10.1145/3463498
10.1109/ACCESS.2020.2967770
10.1109/JIOT.2018.2860592
10.24963/ijcai.2021/631
10.1145/3411841
10.1145/3485730.3485937
10.1007/978-3-319-95276-5_2
10.1145/3300061.3300117
10.1145/3414117
10.1109/INFOCOM48880.2022.9796819
10.1109/INFOCOM.2019.8737633
10.1145/3307334.3326109
10.1109/JIOT.2018.2856119
10.1109/INFOCOM41043.2020.9155526
10.1109/ISM46123.2019.00044
10.1145/3293318
10.1109/INFOCOM41043.2020.9155402
10.1145/3384419.3430725
10.1145/3450268.3453537
10.1109/TIFS.2020.3016819
10.1145/3191755
10.1109/INFOCOM.2017.8057208
10.1145/3241539.3241582
10.1145/2639108.2639110
10.1109/INFOCOM41043.2020.9155380
10.1145/3131672.3131693
10.1016/j.smhl.2021.100205
10.1109/BigData.2018.8622141
10.1145/3458864.3467680
10.1016/j.pmcj.2020.101289
10.1109/TMC.2019.2962760
10.1145/3301275.3302296
10.1109/TMC.2020.3038303
10.1109/INFOCOM.2018.8485958
10.1145/3596255
10.1145/2906388.2906407
10.1145/3381010
10.1109/THMS.2015.2406692
ContentType Magazine Article
DBID 97E
RIA
RIE
AAYXX
CITATION
DOI 10.1109/TMC.2024.3350111
DatabaseName IEEE All-Society Periodicals Package (ASPP) 2005-present
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library Online
CrossRef
DatabaseTitle CrossRef
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library Online
  url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISSN 1558-0660
EndPage 8648
ExternalDocumentID 10_1109_TMC_2024_3350111
10381803
Genre orig-research
GrantInformation_xml – fundername: CityU SRG-Fd
  grantid: 7005658
– fundername: GRF from Research Grants Council of Hong Kong
  grantid: CityU 21201420; CityU 11201422
– fundername: GRF from Research Grants Council of Hong Kong
  grantid: CityU 11217420; CityU 11213622
GroupedDBID -~X
.DC
0R~
1OL
29I
4.4
5GY
5VS
6IK
97E
AAJGR
AASAJ
AAYOK
ABQJQ
ACGFO
ACGFS
ACIWK
AENEX
AETIX
AIBXA
AKJIK
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
DU5
EBS
EJD
HZ~
H~9
IEDLZ
IFIPE
IPLJI
JAVBF
LAI
M43
O9-
OCL
P2P
PQQKQ
RIA
RIC
RIE
RIG
RNI
RNS
RZB
AAYXX
CITATION
ID FETCH-LOGICAL-c217t-cf3f461bf0e8f557c777d75f12caa124c47c1b4ae708818843bda12e11f0db233
IEDL.DBID RIE
ISSN 1536-1233
IngestDate Wed Aug 14 12:29:55 EDT 2024
Wed Aug 14 05:40:39 EDT 2024
IsPeerReviewed true
IsScholarly true
Issue 9
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c217t-cf3f461bf0e8f557c777d75f12caa124c47c1b4ae708818843bda12e11f0db233
ORCID 0000-0002-2474-2004
0000-0002-3296-3392
0009-0006-2074-9109
0000-0001-9741-5912
0000-0002-3918-4922
PageCount 15
ParticipantIDs ieee_primary_10381803
crossref_primary_10_1109_TMC_2024_3350111
PublicationCentury 2000
PublicationDate 2024-09-01
PublicationDateYYYYMMDD 2024-09-01
PublicationDate_xml – month: 09
  year: 2024
  text: 2024-09-01
  day: 01
PublicationDecade 2020
PublicationTitle IEEE transactions on mobile computing
PublicationTitleAbbrev TMC
PublicationYear 2024
Publisher IEEE
Publisher_xml – name: IEEE
References ref13
ref35
ref12
ref34
ref15
ref37
ref14
ref36
Wang (ref39) 2016
ref53
ref30
ref52
ref11
ref33
ref32
ref17
ref16
ref38
ref19
ref18
Nilsson (ref29) 2017
ref51
ref50
ref24
ref46
ref23
ref45
ref26
ref48
ref25
ref47
ref20
Ramponi (ref31) 2018
ref42
ref41
ref22
ref44
ref21
ref43
ref28
ref27
ref49
ref40
References_xml – ident: ref32
  doi: 10.1145/3472291
– ident: ref47
  doi: 10.1109/INFOCOM.2017.8057209
– ident: ref30
  doi: 10.1145/3463498
– ident: ref14
  doi: 10.1109/ACCESS.2020.2967770
– ident: ref37
  doi: 10.1109/JIOT.2018.2860592
– start-page: 189
  volume-title: Proc. ACM Asia Conf. Comput. Commun. Secur.
  year: 2016
  ident: ref39
  article-title: Friend or foe? your wearable devices reveal your personal pin
  contributor:
    fullname: Wang
– ident: ref45
  doi: 10.24963/ijcai.2021/631
– ident: ref20
  doi: 10.1145/3411841
– ident: ref48
  doi: 10.1145/3485730.3485937
– ident: ref12
  doi: 10.1007/978-3-319-95276-5_2
– ident: ref18
  doi: 10.1145/3300061.3300117
– ident: ref23
  doi: 10.1145/3414117
– year: 2018
  ident: ref31
  article-title: T-CGAN: Conditional generative adversarial network for data augmentation in noisy time series with irregular sampling
  contributor:
    fullname: Ramponi
– ident: ref22
  doi: 10.1109/INFOCOM48880.2022.9796819
– ident: ref40
  doi: 10.1109/INFOCOM.2019.8737633
– ident: ref25
  doi: 10.1145/3307334.3326109
– ident: ref46
  doi: 10.1109/JIOT.2018.2856119
– ident: ref51
  doi: 10.1109/INFOCOM41043.2020.9155526
– year: 2017
  ident: ref29
  article-title: Improving the security of the android pattern lock using biometrics and machine learning
  contributor:
    fullname: Nilsson
– ident: ref34
  doi: 10.1109/ISM46123.2019.00044
– ident: ref42
  doi: 10.1145/3293318
– ident: ref43
  doi: 10.1109/INFOCOM41043.2020.9155402
– ident: ref21
  doi: 10.1145/3384419.3430725
– ident: ref26
  doi: 10.1145/3450268.3453537
– ident: ref53
  doi: 10.1109/TIFS.2020.3016819
– ident: ref28
  doi: 10.1145/3191755
– ident: ref17
  doi: 10.1109/INFOCOM.2017.8057208
– ident: ref35
  doi: 10.1145/3241539.3241582
– ident: ref52
  doi: 10.1145/2639108.2639110
– ident: ref15
  doi: 10.1109/INFOCOM41043.2020.9155380
– ident: ref16
  doi: 10.1145/3131672.3131693
– ident: ref41
  doi: 10.1016/j.smhl.2021.100205
– ident: ref13
  doi: 10.1109/BigData.2018.8622141
– ident: ref27
  doi: 10.1145/3458864.3467680
– ident: ref11
  doi: 10.1016/j.pmcj.2020.101289
– ident: ref50
  doi: 10.1109/TMC.2019.2962760
– ident: ref49
  doi: 10.1145/3301275.3302296
– ident: ref44
  doi: 10.1109/TMC.2020.3038303
– ident: ref24
  doi: 10.1109/INFOCOM.2018.8485958
– ident: ref19
  doi: 10.1145/3596255
– ident: ref36
  doi: 10.1145/2906388.2906407
– ident: ref33
  doi: 10.1145/3381010
– ident: ref38
  doi: 10.1109/THMS.2015.2406692
SSID ssj0018997
Score 1.3878623
Snippet This paper addresses a significant problem in American Sign Language (ASL) translation systems that has been overlooked. Current designs collect excessive...
SourceID crossref
ieee
SourceType Aggregation Database
Publisher
StartPage 8634
SubjectTerms Assistive technologies
Computer science
Gesture recognition
Libraries
Mobile computing
Semantics
Sensors
sign language translation
Urban areas
wearable sensing
Title Enhancing the Applicability of Sign Language Translation
URI https://ieeexplore.ieee.org/document/10381803
Volume 23
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LS8NAEB60J0_1UbG-2IMXD0mz7qabHEuxFLG92EJvYV-pIqQi6UF_vbO7iVRB8BaGBIaZ3czOfjPfANwYeeeI6DR6QIiIq9REsqS48VJuhomhWmau33k2H06X_GGVrppmdd8LY631xWc2do8eyzcbvXVXZQPqYS3H7bkv8jw0a31DBpg4iECO6gbLMNZikkk-WMzGsVMpZg5Go_RHDNoZquJjyqQL81abUEryGm9rFevPX0SN_1b3ELotWzQZhfVwBHu2OoZuO7qBNDv5BLL76tkxbVRrgidAMgowti-U_SCbkjy9rCvy2NxlEh_QQtFcD5aT-8V4GjVDFCKN2UYd6ZKVfEhVmdisTFOhhRBGpOgPLSUGd82FpopLK_B_Q7OMM2VQbiktE6PQjqfQqTaVPQPCuBDaog0lZy4vkswqlMicJ1Izyfpw25q1eAtcGYXPMZK8QBcUzgVF44I-9JzBdt4Ltjr_Q34BB-7zUN11CZ36fWuv8DhQq2u_DL4A2duwCA
link.rule.ids 783,787,799,27937,55086
linkProvider IEEE
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV07T8MwELZQGWAqjyLK0wMLQ9IYO3E6VlWrAm0XWqlb5FcKQkoQSgf49ZztBBUkJLboFEWnOzvn83f3HUI3WtxZIjoFHuA8YDLWgcgJbLyY6STSRInU9jvP5slkyR5W8apuVne9MMYYV3xmQvvosHxdqo29KusRB2tZbs9dOFiniW_X-gYNIHXgnh7VjpahtEElo35vMRuGVqmQWiCNkB9RaGusiosq4zaaN_r4YpLXcFPJUH3-omr8t8IHqN3wReOBXxGHaMcUR6jdDG_A9V4-RumoeLZcG8UawxkQDzyQ7UplP3CZ46eXdYGn9W0mdiHNl8110HI8WgwnQT1GIVCQb1SBymnOEiLzyKR5HHPFOdc8Bo8oISC8K8YVkUwYDn8ckqaMSg1yQ0geaQl2PEGtoizMKcKUca4M2FAwajMjQY0EieizSCgqaBfdNmbN3jxbRuayjKifgQsy64KsdkEXdazBtt7ztjr7Q36N9iaL2TSb3s8fz9G-_ZSv9bpArep9Yy7hcFDJK7ckvgBA97NT
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Enhancing+the+Applicability+of+Sign+Language+Translation&rft.jtitle=IEEE+transactions+on+mobile+computing&rft.au=Li%2C+Jiao&rft.au=Xu%2C+Jiakai&rft.au=Liu%2C+Yang&rft.au=Xu%2C+Weitao&rft.date=2024-09-01&rft.pub=IEEE&rft.issn=1536-1233&rft.volume=23&rft.issue=9&rft.spage=8634&rft.epage=8648&rft_id=info:doi/10.1109%2FTMC.2024.3350111&rft.externalDocID=10381803
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1536-1233&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1536-1233&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1536-1233&client=summon