Mental state analysis system, electronic equipment and storage medium

• Main Author: TAO JIANHUA
• Format: Patent
• Language: Chinese; English
• Published: 01.03.2024
• Subjects: ACOUSTICS; CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; DIAGNOSIS; HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATIONTECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING ORPROCESSING OF MEDICAL OR HEALTHCARE DATA; HUMAN NECESSITIES; HYGIENE; IDENTIFICATION; INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTEDFOR SPECIFIC APPLICATION FIELDS; MEDICAL OR VETERINARY SCIENCE; MUSICAL INSTRUMENTS; PHYSICS; SPEECH ANALYSIS OR SYNTHESIS; SPEECH OR AUDIO CODING OR DECODING; SPEECH OR VOICE PROCESSING; SPEECH RECOGNITION; SURGERY

The invention provides a mental state analysis system, electronic equipment and a storage medium, and the system comprises a multi-modal feature extraction module, a cross-modal attention module and a mental state detection module. The multi-modal feature extraction module converts long audio and video into an image sequence and a voice segment, and extracts facial features and audio features; inputting the facial features and the audio features into the cross-modal attention module, exploring a relationship between an image and an audio modal, and extracting an emotion feature value of the subject; and inputting the emotion feature value into a mental state detection module to obtain the current mental state of the subject. According to the scheme provided by the invention, a single cross-modal attention module can be used for extracting the mental state characteristic values of the audio and video characteristics, the characteristics are used for detecting and analyzing depression, anxiety and fatigue state