Sample generation method and device, model training method and device, mouth shape driving method and device, equipment and medium

• Main Authors: HAN XINTONG, LIU FENG, TUO DEYI, LIN ZHE, XU JIA, YOU YUREN, SI JIYU, KANG SHIYIN
• Format: Patent
• Language: Chinese; English
• Published: 27.08.2021
• Subjects: ACOUSTICS; CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; MUSICAL INSTRUMENTS; PHYSICS; SPEECH ANALYSIS OR SYNTHESIS; SPEECH OR AUDIO CODING OR DECODING; SPEECH OR VOICE PROCESSING; SPEECH RECOGNITION

The embodiment of the invention discloses a sample generation method and device, a model training method and device, a mouth shape driving method and device, equipment and a medium. The sample generation method comprises the following steps: capturing a voice segment and a mouth shape driving parameter sequence formed by a real person in a speaking process; according to the driving result, performing time alignment on the mouth shape driving parameter sequence and the voice segment; identifying starting and ending time of each effective voice segment in the voice segment, and determining mouth shape driving parameter subsequence corresponding to each effective voice segment in the mouth shape driving parameter sequence according to the starting and ending time; and according to the content representation vectors corresponding to the effective voice segments and the mouth shape driving parameter subsequences corresponding to the effective voice segments, forming training sample pairs for training to obtain a m