Towards Ultrasound Tongue Image prediction from EEG during speech production
Previous initial research has already been carried out to propose speech-based BCI using brain signals (e.g. non-invasive EEG and invasive sEEG / ECoG), but there is a lack of combined methods that investigate non-invasive brain, articulation, and speech signals together and analyze the cognitive pr...
Saved in:
Published in | arXiv.org |
---|---|
Main Authors | , , , |
Format | Paper Journal Article |
Language | English |
Published |
Ithaca
Cornell University Library, arXiv.org
18.10.2023
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Previous initial research has already been carried out to propose speech-based BCI using brain signals (e.g. non-invasive EEG and invasive sEEG / ECoG), but there is a lack of combined methods that investigate non-invasive brain, articulation, and speech signals together and analyze the cognitive processes in the brain, the kinematics of the articulatory movement and the resulting speech signal. In this paper, we describe our multimodal (electroencephalography, ultrasound tongue imaging, and speech) analysis and synthesis experiments, as a feasibility study. We extend the analysis of brain signals recorded during speech production with ultrasound-based articulation data. From the brain signal measured with EEG, we predict ultrasound images of the tongue with a fully connected deep neural network. The results show that there is a weak but noticeable relationship between EEG and ultrasound tongue images, i.e. the network can differentiate articulated speech and neutral tongue position. |
---|---|
AbstractList | Previous initial research has already been carried out to propose speech-based BCI using brain signals (e.g. non-invasive EEG and invasive sEEG / ECoG), but there is a lack of combined methods that investigate non-invasive brain, articulation, and speech signals together and analyze the cognitive processes in the brain, the kinematics of the articulatory movement and the resulting speech signal. In this paper, we describe our multimodal (electroencephalography, ultrasound tongue imaging, and speech) analysis and synthesis experiments, as a feasibility study. We extend the analysis of brain signals recorded during speech production with ultrasound-based articulation data. From the brain signal measured with EEG, we predict ultrasound images of the tongue with a fully connected deep neural network. The results show that there is a weak but noticeable relationship between EEG and ultrasound tongue images, i.e. the network can differentiate articulated speech and neutral tongue position. Proceedings of Interspeech 2023 Previous initial research has already been carried out to propose speech-based BCI using brain signals (e.g. non-invasive EEG and invasive sEEG / ECoG), but there is a lack of combined methods that investigate non-invasive brain, articulation, and speech signals together and analyze the cognitive processes in the brain, the kinematics of the articulatory movement and the resulting speech signal. In this paper, we describe our multimodal (electroencephalography, ultrasound tongue imaging, and speech) analysis and synthesis experiments, as a feasibility study. We extend the analysis of brain signals recorded during speech production with ultrasound-based articulation data. From the brain signal measured with EEG, we predict ultrasound images of the tongue with a fully connected deep neural network. The results show that there is a weak but noticeable relationship between EEG and ultrasound tongue images, i.e. the network can differentiate articulated speech and neutral tongue position. |
Author | Boncz, Ádám Frigyes Viktor Arthur Tamás Gábor Csapó Nagy, Péter |
Author_xml | – sequence: 1 fullname: Tamás Gábor Csapó – sequence: 2 fullname: Frigyes Viktor Arthur – sequence: 3 givenname: Péter surname: Nagy fullname: Nagy, Péter – sequence: 4 givenname: Ádám surname: Boncz fullname: Boncz, Ádám |
BackLink | https://doi.org/10.48550/arXiv.2306.05374$$DView paper in arXiv https://doi.org/10.21437/Interspeech.2023-40$$DView published paper (Access to full text may be restricted) |
BookMark | eNotj8tOwzAURC0EEqX0A1hhiXXC9bUdO0tUhVKpEpuwjkzslFStXeyGx983tKxmczQz54Zc-uAdIXcMcqGlhEcTf_qvHDkUOUiuxAWZIOcs0wLxmsxS2gAAFgql5BOyqsO3iTbRt-0hmhQGb2kd_HpwdLkza0f30dm-PfTB0y6GHa2qBbVD7P2apr1z7cdIBDuciFty1ZltcrP_nJL6uarnL9nqdbGcP60yU0qR8baAEjuhC6lRtMZZFJqBckKBVkxahaB0-16ygllURhsH0pact9wBQ-RTcn-uPak2-9jvTPxt_pSbk_JIPJyJ8dvn4NKh2YQh-vFTg-MmE4WWJT8CbcRZsA |
ContentType | Paper Journal Article |
Copyright | 2023. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. http://creativecommons.org/licenses/by/4.0 |
Copyright_xml | – notice: 2023. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. – notice: http://creativecommons.org/licenses/by/4.0 |
DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PIMPY PQEST PQQKQ PQUKI PRINS PTHSS AKY GOX |
DOI | 10.48550/arxiv.2306.05374 |
DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest Central ProQuest Central Essentials ProQuest Central Technology Collection ProQuest One Community College ProQuest Central SciTech Premium Collection ProQuest Engineering Collection Engineering Database ProQuest - Publicly Available Content Database ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection arXiv Computer Science arXiv.org |
DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest One Academic Engineering Collection |
DatabaseTitleList | Publicly Available Content Database |
Database_xml | – sequence: 1 dbid: GOX name: arXiv.org url: http://arxiv.org/find sourceTypes: Open Access Repository – sequence: 2 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Physics |
EISSN | 2331-8422 |
ExternalDocumentID | 2306_05374 |
Genre | Working Paper/Pre-Print |
GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PIMPY PQEST PQQKQ PQUKI PRINS PTHSS AKY GOX |
ID | FETCH-LOGICAL-a954-3c6092f4865824caed248107e4708715d72078cb9161d27a8ae05d933c3e01223 |
IEDL.DBID | BENPR |
IngestDate | Mon Jan 08 05:38:23 EST 2024 Thu Oct 10 18:27:10 EDT 2024 |
IsDoiOpenAccess | true |
IsOpenAccess | true |
IsPeerReviewed | false |
IsScholarly | false |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-a954-3c6092f4865824caed248107e4708715d72078cb9161d27a8ae05d933c3e01223 |
OpenAccessLink | https://www.proquest.com/docview/2824146859?pq-origsite=%requestingapplication% |
PQID | 2824146859 |
PQPubID | 2050157 |
ParticipantIDs | arxiv_primary_2306_05374 proquest_journals_2824146859 |
PublicationCentury | 2000 |
PublicationDate | 20231018 |
PublicationDateYYYYMMDD | 2023-10-18 |
PublicationDate_xml | – month: 10 year: 2023 text: 20231018 day: 18 |
PublicationDecade | 2020 |
PublicationPlace | Ithaca |
PublicationPlace_xml | – name: Ithaca |
PublicationTitle | arXiv.org |
PublicationYear | 2023 |
Publisher | Cornell University Library, arXiv.org |
Publisher_xml | – name: Cornell University Library, arXiv.org |
SSID | ssj0002672553 |
Score | 1.9025743 |
SecondaryResourceType | preprint |
Snippet | Previous initial research has already been carried out to propose speech-based BCI using brain signals (e.g. non-invasive EEG and invasive sEEG / ECoG), but... Proceedings of Interspeech 2023 Previous initial research has already been carried out to propose speech-based BCI using brain signals (e.g. non-invasive EEG... |
SourceID | arxiv proquest |
SourceType | Open Access Repository Aggregation Database |
SubjectTerms | Artificial neural networks Brain Computer Science - Sound Electroencephalography Feasibility studies Kinematics Physics - Medical Physics Speech Tongue |
SummonAdditionalLinks | – databaseName: arXiv.org dbid: GOX link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwdV07T8MwELbaTiwIBKiFgjywBlLHjpOxQn2AeCyp1C3yK4AEaZS0iJ_PnZOKAbFF0Xn5HN93l_N9R8i1nVirjY0DYJMo4LHQAdAIPLkExT5CqRT-73h6jpcr_rAW6x6h-14YVX-_f7X6wLq5xfj4BhVHeJ_0GcMrW4uXdVuc9FJcnf2vHcSY_tUf1-r5Yn5EDrtAj07bnTkmPVeekMfM31Jt6OpjW6sGRxrRbFO-7hy9_4STTasaCycIFsXGDzqbLWjbSUibyjnzRqtWohUsTkk2n2V3y6CbZxCoVPAgMnGYsoInQPqMG-Us4wlkX47LENIWYSUDvjYaAraJZVIlyoXCplFkIocFsOiMDMpN6YaEFoWWWjBwTRIOnWapltIA-bKC2TROwhEZehTyqpWsyBGg3AM0IuM9MHn3uTY55F0ce7BEev7_ygtygLPW0XFPkjEZbOuduwRG3uorvy0_zUqJwA priority: 102 providerName: Cornell University |
Title | Towards Ultrasound Tongue Image prediction from EEG during speech production |
URI | https://www.proquest.com/docview/2824146859 https://arxiv.org/abs/2306.05374 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV3NT4MwFG_cFhNvfmbTufTgFcdKoXAy0bBN4-ZiWLIbKW2nJsoQNuPJv933gOnBxAsByumV_n7v-xFyoQdaJ0p7FrCJY3HPTSygEbgzPjb7sIWU6O-YTL3xnN8t3EXtcCvqtMotJpZArVcKfeR9MA04lgm5wVX2buHUKIyu1iM0GqTFBhzDtK3rcDp7_PGyME-AzuxU4cyyeVdf5p8vH5j_7F1iLxMOWmn56g8Ylwwz3CetmcxMfkB2THpIdsvETFUckfuozGst6Px1ncsChyDRaJU-bQy9fQMsoFmOoRYUL8VSERqGI1rVHtIiM0Y906xq6gpfHJNoGEY3Y6uegGDJwOWWozw7YEvug5rAuJJGM-6DvWa4sMHQcbVgwPAqARVvoJmQvjS2qwPHUY7BkJlzQprpKjVtQpfLRCQuAzATcEwTFiRCKKBrtmQ68Hy7Q9qlFOKsanIRo4DiUkAd0t0KJq5_8CL-3Y7T_5fPyB5OaEe4H_hd0lznG3MOPL5OeqThD0e9esvgafSwgOvkK_wGF_ee9w |
link.rule.ids | 228,230,786,790,891,12792,21416,27956,33406,33777,43633,43838 |
linkProvider | ProQuest |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV07T8MwELagFYKNp1oo4IE1NLWdOJkYUF_QVgyp1C1ybBeQoA1Ji_j53LkpDEhsVpzpbH_fvY-QG9MxJtMm9IBNuCfCIPOARmBlI2z24Uul0N8xnoSDqXiYBbPK4VZWaZVbTHRAbZYafeRtMA0ElgkF8V3-4eHUKIyuViM0dkld8JDjPY96_R8fCwslaMx8E8x0rbvaqvh6_cTs5_AWO5kI0Endpz9Q7Pild0jqTyq3xRHZsYtjsufSMnV5QkaJy2ot6fRtVagSRyDRZLl4Xls6fAckoHmBgRYULsVCEdrt9umm8pCWubX6heablq7wxylJet3kfuBV8w88FQfC4zr0YzYXESgJTGhlDRMRWGtWSB_MnMBIBvyuM1DwOoZJFSnrBybmXHOLATN-RmqL5cI2CJ3PM5kFDKBMwiPNWJxJqYGs2ZyZOIz8Jmk4KaT5psVFigJKnYCapLUVTFpd7zL9PYzz_7evyf4gGY_S0XDyeEEOcFY7An8napHaqljbS2D0VXblju0bxmSdWA |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+Ultrasound+Tongue+Image+prediction+from+EEG+during+speech+production&rft.jtitle=arXiv.org&rft.au=Tam%C3%A1s+G%C3%A1bor+Csap%C3%B3&rft.au=Frigyes+Viktor+Arthur&rft.au=Nagy%2C+P%C3%A9ter&rft.au=Boncz%2C+%C3%81d%C3%A1m&rft.date=2023-10-18&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422&rft_id=info:doi/10.48550%2Farxiv.2306.05374 |