Deformed landmark fitting for sequential faces

• Published in: Journal of visual communication and image representation Vol. 62; pp. 381 - 393
• Main Authors: Han, Shoudong, Yang, Ziqing, Li, Qianqian, Chen, Yang
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.07.2019
• Subjects: Active shape model; Global shape model; Landmark fitting; Large displacement optical flow model; Landmark fitting; Active shape model; Large displacement optical flow model; Global shape model
• ISSN: 1047-3203; 1095-9076
• DOI: 10.1016/j.jvcir.2019.06.010

Fitting facial landmarks on unconstrained videos is a challenging task with broad applications. At present, many methods of one-shot landmark fitting have been proposed with varying degrees of success. However, most of them are heavily sensitive to initializations and usually rely on offline-trained static models, which limit their performance on sequential images with extensive variations. Therefore, they usually can’t align the deformed face very well. To address these limitations, we propose a method of deformed landmark fitting (DLF) for sequential faces, which is designed based on active shape model (ASM) and deformation tracking/correction. This method overcomes the loss of consecutive information between frames, and makes full use of the motion variation information of video sequences in time and space dimensions. Firstly, the optical flow values of several possible deformation points on the face are calculated by the large displacement optical flow (LDOF) model, and the tracking of these points in the current frame are performed through the optical flow motion vector. Secondly, the initial shape of face in each frame is established by the locations of these deformation points and the global shape model in ASM algorithm. Finally, on the basis of initial shape, according to the guidance of local texture model in ASM algorithm, different correction strategies are applied to different landmarks for local search, and then each landmark is reasonably suppressed to obtain the ultimate results. Our DLF observably improves the fitting accuracy for deformed faces, and takes full advantage of the continuity among video sequences. Compared with some state-of-the-art landmarkers, extensive experiments on landmark fitting for sequential faces show that our DLF performs outstandingly in terms of accuracy and robustness.