Gesture recognition method and device based on voice and video, equipment and medium

• Main Authors: LI CAIBO, GENG CHANGBIAO, WEN JINHONG, MA HAN
• Format: Patent
• Language: Chinese; English
• Published: 20.10.2023
• Subjects: ACOUSTICS; CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; MUSICAL INSTRUMENTS; PHYSICS; SPEECH ANALYSIS OR SYNTHESIS; SPEECH OR AUDIO CODING OR DECODING; SPEECH OR VOICE PROCESSING; SPEECH RECOGNITION

The invention provides a gesture recognition method and device based on voice and video, equipment and a medium, and relates to the field of data processing, and the method comprises the steps: obtaining video data and voice data, and carrying out the preprocessing of the video data and the voice data, and obtaining first processing data and second processing data; establishing an initial model, and training by adopting the training data to obtain a target model; wherein the target model comprises a first feature extraction network and a second feature extraction network formed by a plurality of stacked convolution blocks, and a feature fusion module formed by a plurality of convolution layers; performing image feature extraction on the first processing data through the first feature extraction network to obtain a first feature; performing voice feature extraction on the second processing data through the second feature extraction network to obtain a second feature; and based on the first feature and the seco