Building a Large Database of Facial Movements for Deformation Model‐Based 3D Face Tracking

• Published in: Computer graphics forum Vol. 36; no. 8; pp. 285 - 301
• Main Authors: Sibbing, Dominik, Kobbelt, Leif
• Format: Journal Article
• Language: English
• Published: Oxford Blackwell Publishing Ltd 01.12.2017
• Subjects: Animation; Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Three‐Dimensional Graphics and Realism; data‐driven animation; Deformation; Face recognition; facial animation; Formability; I.4.8 [Image Processing and Computer Vision]: Scene Analysis; markerless performance capture; Principal components analysis; Sequences; Smoothness; Three dimensional models; tracking
• ISSN: 0167-7055; 1467-8659
• DOI: 10.1111/cgf.13080

We introduce a new markerless 3D face tracking approach for 2D videos captured by a single consumer grade camera. Our approach takes detected 2D facial features as input and matches them with projections of 3D features of a deformable model to determine its pose and shape. To make the tracking and reconstruction more robust we add a smoothness prior for pose and deformation changes of the faces. Our major contribution lies in the formulation of the deformation prior which we derive from a large database of facial animations showing different (dynamic) facial expressions of a fairly large number of subjects. We split these animation sequences into snippets of fixed length which we use to predict the facial motion based on previous frames. In order to keep the deformation model compact and independent from the individual physiognomy, we represent it by deformation gradients (instead of vertex positions) and apply a principal component analysis in deformation gradient space to extract the major modes of facial deformation. Since the facial deformation is optimized during tracking, it is particularly easy to apply them to other physiognomies and thereby re‐target the facial expressions. We demonstrate the effectiveness of our technique on a number of examples. We introduce a new markerless 3D face tracking approach for 2D videos captured by a single consumer grade camera. Our approach takes detected 2D facial features as input and matches them with projections of 3D features of a deformable model to determine its pose and shape. To make the tracking and reconstruction more robust we add a smoothness prior for pose and deformation changes of the faces. Our major contribution lies in the formulation of the deformation prior which we derive from a large database of facial animations showing different (dynamic) facial expressions of a fairly large number of subjects. We split these animation sequences into snippets of fixed length which we use to predict the facial motion based on previous frames.