基于跨模态语义信息增强的多模态情感分析
TP391; 随着社交网络的发展,人类通过不同的方式表达自己的情感,包括文本、视觉和语音,即多模态.针对以往的多模态情感分析方法未能有效地获取多模态情感特征表示,以及没有充分考虑在多模态特征融合过程中冗余信息对实验的影响,提出了一种基于跨模态语义信息增强的多模态情感分析模型.该模型采用BiLSTM网络挖掘各单模态内部存在的上下文信息.通过跨模态信息交互机制对多种模态间的信息交互进行建模,得到文本对语音、视觉,语音对文本、视觉,视觉对文本、语音六种信息交互特征,将目标模态相同的信息交互特征进行拼接,得到信息增强后的单模态特征向量,有效地获取模态间共享和补充的深度语义特征.另外,使用多头自注意力机...
Saved in:
| Published in | 计算机科学与探索 Vol. 18; no. 9; pp. 2476 - 2486 |
|---|---|
| Main Authors | , , , , |
| Format | Journal Article |
| Language | Chinese |
| Published |
内蒙古科技大学 信息工程学院,内蒙古 包头 014010%内蒙古科技大学 理学院,内蒙古 包头 014010%内蒙古科技大学 创新创业教育学院,内蒙古 包头 014010
01.09.2024
上海大学 计算机工程与科学学院,上海 200444 |
| Subjects | |
| Online Access | Get full text |
| ISSN | 1673-9418 |
| DOI | 10.3778/j.issn.1673-9418.2307045 |
Cover
| Abstract | TP391; 随着社交网络的发展,人类通过不同的方式表达自己的情感,包括文本、视觉和语音,即多模态.针对以往的多模态情感分析方法未能有效地获取多模态情感特征表示,以及没有充分考虑在多模态特征融合过程中冗余信息对实验的影响,提出了一种基于跨模态语义信息增强的多模态情感分析模型.该模型采用BiLSTM网络挖掘各单模态内部存在的上下文信息.通过跨模态信息交互机制对多种模态间的信息交互进行建模,得到文本对语音、视觉,语音对文本、视觉,视觉对文本、语音六种信息交互特征,将目标模态相同的信息交互特征进行拼接,得到信息增强后的单模态特征向量,有效地获取模态间共享和补充的深度语义特征.另外,使用多头自注意力机制分别计算原始单模态特征向量和信息增强后的单模态特征向量间存在的语义相关性,提高识别关键情感特征的能力,降低冗余信息对情感分析的负面干扰.在公共数据集CMU-MOSI和CMU-MOSEI的实验结果表明,所提出的模型既能增强情感特征表示,也能有效降低冗余信息的干扰,在多模态情感分类准确率和泛化能力上的表现优于相关工作. |
|---|---|
| AbstractList | TP391; 随着社交网络的发展,人类通过不同的方式表达自己的情感,包括文本、视觉和语音,即多模态.针对以往的多模态情感分析方法未能有效地获取多模态情感特征表示,以及没有充分考虑在多模态特征融合过程中冗余信息对实验的影响,提出了一种基于跨模态语义信息增强的多模态情感分析模型.该模型采用BiLSTM网络挖掘各单模态内部存在的上下文信息.通过跨模态信息交互机制对多种模态间的信息交互进行建模,得到文本对语音、视觉,语音对文本、视觉,视觉对文本、语音六种信息交互特征,将目标模态相同的信息交互特征进行拼接,得到信息增强后的单模态特征向量,有效地获取模态间共享和补充的深度语义特征.另外,使用多头自注意力机制分别计算原始单模态特征向量和信息增强后的单模态特征向量间存在的语义相关性,提高识别关键情感特征的能力,降低冗余信息对情感分析的负面干扰.在公共数据集CMU-MOSI和CMU-MOSEI的实验结果表明,所提出的模型既能增强情感特征表示,也能有效降低冗余信息的干扰,在多模态情感分类准确率和泛化能力上的表现优于相关工作. |
| Abstract_FL | With the development of social networks,humans express their emotions in different ways,including text,vision and speech,i.e.,multimodal.In response to the failure of previous multimodal sentiment analysis methods to effectively obtain multimodal sentiment feature representations and the failure to fully consider the impact of redundant information on experiments during multimodal feature fusion,a multimodal sentiment analysis model based on cross-modal semantic information enhancement is proposed.Firstly,the model adopts BiLSTM network to mine the contextual information within each unimodal mode.Secondly,the information interaction between multiple modalities is modeled through the cross-modal information interaction mechanism to obtain six kinds of information interaction features,namely,text-to-speech and vision,speech-to-text and vision,and vision-to-text and speech,and then the same information interaction features of the target modalities are spliced together to obtain the information-enhanced unimodal feature vectors,which can efficiently obtain the shared and complementary in-depth semantic features between modalities.In addition,the semantic correlations between the original unimodal feature vectors and the information-enhanced unimodal feature vectors are computed separately using the multi-head self-attention mechanism,which improves the ability of identifying the key sentiment features and reduces the negative interference of the redundant information on the sentiment analysis.Experimental results on the public datasets CMU-MOSI(CMU multimodal opinion level sentiment intensity)and CMU-MOSEI(CMU multimodal opinion sentiment and emotion intensity)show that the proposed model can both enhance sentiment feature representation and effectively reduce the interference of redundant information,and it outperforms related works in terms of multimodal sentiment classification accuracy and generalization ability. |
| Author | 张晓琳 李梦云 张景 刘璐瑶 张换香 |
| AuthorAffiliation | 内蒙古科技大学 信息工程学院,内蒙古 包头 014010%内蒙古科技大学 理学院,内蒙古 包头 014010%内蒙古科技大学 创新创业教育学院,内蒙古 包头 014010;上海大学 计算机工程与科学学院,上海 200444 |
| AuthorAffiliation_xml | – name: 内蒙古科技大学 信息工程学院,内蒙古 包头 014010%内蒙古科技大学 理学院,内蒙古 包头 014010%内蒙古科技大学 创新创业教育学院,内蒙古 包头 014010;上海大学 计算机工程与科学学院,上海 200444 |
| Author_FL | LI Mengyun ZHANG Huanxiang ZHANG Xiaolin ZHANG Jing LIU Luyao |
| Author_FL_xml | – sequence: 1 fullname: LI Mengyun – sequence: 2 fullname: ZHANG Jing – sequence: 3 fullname: ZHANG Huanxiang – sequence: 4 fullname: ZHANG Xiaolin – sequence: 5 fullname: LIU Luyao |
| Author_xml | – sequence: 1 fullname: 李梦云 – sequence: 2 fullname: 张景 – sequence: 3 fullname: 张换香 – sequence: 4 fullname: 张晓琳 – sequence: 5 fullname: 刘璐瑶 |
| BookMark | eNo9jbtKA0EYhaeIYIx5B1uLXf-57cyUErxBwEbrsLuZkawyAUdRO4NRtBHrFAmRFGlCSnERfJndMY9hIGJ14HyH72ygiu1ajdAWhpAKIXeysOOcDXEkaKAYliGhIIDxCqr-d-uo7lwnAc4YwSKSVaTKUV7kr4uPqZ-O_X1vMZ8Vny_F99j35uX7sPzKfwb9cjJYUf_w6Puj8vnJD9820ZqJL5yu_2UNne7vnTQOg-bxwVFjtxk4DEwFxnBqYsGYaicJA5mmWJt2olOqjQAdERFDREEtJ5wrIhTnONEilURJaQihNbS98t7E1sT2rJV1ry_t8rGVuez89u7KESAMFGBFfwH6UmFc |
| ClassificationCodes | TP391 |
| ContentType | Journal Article |
| Copyright | Copyright © Wanfang Data Co. Ltd. All Rights Reserved. |
| Copyright_xml | – notice: Copyright © Wanfang Data Co. Ltd. All Rights Reserved. |
| DBID | 2B. 4A8 92I 93N PSX TCJ |
| DOI | 10.3778/j.issn.1673-9418.2307045 |
| DatabaseName | Wanfang Data Journals - Hong Kong WANFANG Data Centre Wanfang Data Journals 万方数据期刊 - 香港版 China Online Journals (COJ) China Online Journals (COJ) |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| DocumentTitle_FL | Multimodal Sentiment Analysis Based on Cross-Modal Semantic Information Enhancement |
| EndPage | 2486 |
| ExternalDocumentID | jsjkxyts202409019 |
| GroupedDBID | 2B. 4A8 92I 93N ALMA_UNASSIGNED_HOLDINGS M~E PSX TCJ |
| ID | FETCH-LOGICAL-s1049-ff53fa7449dbb408cc1efdbec3ef70e627a06309a74559279551be7c82988f223 |
| ISSN | 1673-9418 |
| IngestDate | Thu May 29 04:00:18 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 9 |
| Keywords | 多模态情感分析 特征融合 feature fusion multimodal emotional analysis multi-head attention mechanism information interaction 信息增强 多头注意力机制 信息交互 information augmentation |
| Language | Chinese |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-s1049-ff53fa7449dbb408cc1efdbec3ef70e627a06309a74559279551be7c82988f223 |
| PageCount | 11 |
| ParticipantIDs | wanfang_journals_jsjkxyts202409019 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-09-01 |
| PublicationDateYYYYMMDD | 2024-09-01 |
| PublicationDate_xml | – month: 09 year: 2024 text: 2024-09-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationTitle | 计算机科学与探索 |
| PublicationTitle_FL | Journal of Frontiers of Computer Science & Technology |
| PublicationYear | 2024 |
| Publisher | 内蒙古科技大学 信息工程学院,内蒙古 包头 014010%内蒙古科技大学 理学院,内蒙古 包头 014010%内蒙古科技大学 创新创业教育学院,内蒙古 包头 014010 上海大学 计算机工程与科学学院,上海 200444 |
| Publisher_xml | – name: 上海大学 计算机工程与科学学院,上海 200444 – name: 内蒙古科技大学 信息工程学院,内蒙古 包头 014010%内蒙古科技大学 理学院,内蒙古 包头 014010%内蒙古科技大学 创新创业教育学院,内蒙古 包头 014010 |
| SSID | ssib054421768 ssib002040941 ssib002423894 ssib051375751 ssib023646573 ssib036438069 ssib002040926 |
| Score | 2.3720706 |
| Snippet | TP391;... |
| SourceID | wanfang |
| SourceType | Aggregation Database |
| StartPage | 2476 |
| Title | 基于跨模态语义信息增强的多模态情感分析 |
| URI | https://d.wanfangdata.com.cn/periodical/jsjkxyts202409019 |
| Volume | 18 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwzR3LbtNA0CrthQsC8X4JIfZUufix3t052o2jCqmcWqm3yk5sUJGCRBIJekBUFAQXxLmHVkUcekE9IiokfqYJ_Qxmxk5i2hSVx4HLZrw7OzuPtXfG2Vlb1h28_f1m7kg7yZSxJSjXNm6Q24mRppE3crzJKDl5_r6aW5T3loKlialLlV1L3U4601gdm1fyJ1bFOrQrZcn-hmWHRLECYbQvlmhhLE9kYxEHAuoiCkUsqTSxiI2ItAiNiBWVoUuAcdBhpKawLsIaI4MwwEB9gONSKxIMPQExAdEsU9YCkLLkJknwGMoI-MIEDEhiCZGNEUZRDVIrvtU5cIKZk5gpaAJAM9pguBBHdHm4mggVM2lYNEUlsoc4kURgMFm4d61EwfayUyjAHaGwPKHDuICijm0xNaYPRALguM7gs1YcEfnVlyaeHO4KK6Y5a0GxXowAj7pSTZ30OJITEULSY6FfrBxIPj3ePmTegPujVqKKnoAF8_Cm-cXA0wzNchsPGMlpDn8d4QV_zzCrhcz-X3HFkxEiNp8SkVOp4clVTGoImDRq1ROR9-9EqKz7Svs2yNIVOOIYdAepD-UqL7WqeIyeLE5zP-yN-Fob9kZoiJnhEJx74RSHqB46632lvfLo6bNOm6asg546nLKmPK15_8n883jkV-PSC9X3AnQtf0pQx0BkuNDSRyJUMIpT8NI3jhrGMYHra_r_dXgtpefqIo14wHWxC5JEunucQJz32MqT1oOKi75w1jpTxta3wuJBec6aWH143oLe1t7-3ruDzzv9ne3-i7WD3U_7X97uf9vur-32Pmz2vu5931jvfdwoWvsvX_XXt3pvXvc331-wFuvxwuycXX4vxm67jgQ7zwM_T7SU0ExT6ZhGw83yJi5SfpZrJ1OeTuiEQUCUIABPA0aLaaYbxgNjcoyTLlqTrcet7DKKioGwzBOZBj7gbwJplpO7oiDzZNN1r1i3S1GXy_WgvXzEeFdPgnTNOj16QF23JjtPutkNjHM66U22-Q8ct-fB |
| linkProvider | ISSN International Centre |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%E5%9F%BA%E4%BA%8E%E8%B7%A8%E6%A8%A1%E6%80%81%E8%AF%AD%E4%B9%89%E4%BF%A1%E6%81%AF%E5%A2%9E%E5%BC%BA%E7%9A%84%E5%A4%9A%E6%A8%A1%E6%80%81%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90&rft.jtitle=%E8%AE%A1%E7%AE%97%E6%9C%BA%E7%A7%91%E5%AD%A6%E4%B8%8E%E6%8E%A2%E7%B4%A2&rft.au=%E6%9D%8E%E6%A2%A6%E4%BA%91&rft.au=%E5%BC%A0%E6%99%AF&rft.au=%E5%BC%A0%E6%8D%A2%E9%A6%99&rft.au=%E5%BC%A0%E6%99%93%E7%90%B3&rft.date=2024-09-01&rft.pub=%E5%86%85%E8%92%99%E5%8F%A4%E7%A7%91%E6%8A%80%E5%A4%A7%E5%AD%A6+%E4%BF%A1%E6%81%AF%E5%B7%A5%E7%A8%8B%E5%AD%A6%E9%99%A2%2C%E5%86%85%E8%92%99%E5%8F%A4+%E5%8C%85%E5%A4%B4+014010%25%E5%86%85%E8%92%99%E5%8F%A4%E7%A7%91%E6%8A%80%E5%A4%A7%E5%AD%A6+%E7%90%86%E5%AD%A6%E9%99%A2%2C%E5%86%85%E8%92%99%E5%8F%A4+%E5%8C%85%E5%A4%B4+014010%25%E5%86%85%E8%92%99%E5%8F%A4%E7%A7%91%E6%8A%80%E5%A4%A7%E5%AD%A6+%E5%88%9B%E6%96%B0%E5%88%9B%E4%B8%9A%E6%95%99%E8%82%B2%E5%AD%A6%E9%99%A2%2C%E5%86%85%E8%92%99%E5%8F%A4+%E5%8C%85%E5%A4%B4+014010&rft.issn=1673-9418&rft.volume=18&rft.issue=9&rft.spage=2476&rft.epage=2486&rft_id=info:doi/10.3778%2Fj.issn.1673-9418.2307045&rft.externalDocID=jsjkxyts202409019 |
| thumbnail_s | http://utb.summon.serialssolutions.com/2.0.0/image/custom?url=http%3A%2F%2Fwww.wanfangdata.com.cn%2Fimages%2FPeriodicalImages%2Fjsjkxyts%2Fjsjkxyts.jpg |