Speech-based three-dimensional facial animation

• Main Authors: FERNANDO DE LA TORRE, SOLLHOFER, MICHAEL, SHAIKH, YASER, RICHARD ALEXANDRE
• Format: Patent
• Language: Chinese; English
• Published: 03.11.2023
• Subjects: CALCULATING; COMPUTING; COUNTING; IMAGE DATA PROCESSING OR GENERATION, IN GENERAL; PHYSICS

A method for speech-based training of a three-dimensional model facial animation model is provided. The method includes: determining a first correlation value of a facial feature based on an audio waveform from a first object; generating a first grid for the lower part of the face based on the facial feature and the first correlation value; updating the first correlation value when the difference between the first grid and the ground truth value image of the first object is greater than a pre-selected threshold value; and providing the three-dimensional model of the face animated by the voice to an immersive reality application accessed by the client device based on a difference between the first mesh and the ground truth image of the first object. A non-transitory computer readable medium and a system are also provided. The non-transitory computer readable medium stores instructions to cause the system to perform the above method. 提供了一种用于基于语音训练三维模型面部动画模型的方法。该方法包括：基于来自第一对象的音频波形确定面部特征的第一相关值；基于面部特征和第一相关值，为人脸的下部