Deepfacelab: Integrated, flexible and extensible face-swapping framework

• Published in: Pattern recognition Vol. 141; p. 109628
• Main Authors: Liu, Kunlin, Perov, Ivan, Gao, Daiheng, Chervoniy, Nikolay, Zhou, Wenbo, Zhang, Weiming
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.09.2023
• Subjects: Face swapping; Open source; Practical machine learning; Practical machine learning; Open source; Face swapping
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2023.109628

•Recent researches on face swapping methods focus on adjusting the identity extraction methods on one-shot scenarios. However, no matter how these models are adjusted, the identity information is always obtained from one shot source image. In other words, the identity information these methods leverage are guessed by a prior model. To achieve better performance, we develop an integrated, flexible and extensible framework, DeepFaceLab, to conduct cinema-level face-swapping with multi-shot data.•To make our framework practical, we invente a lot of auxiliary tools in each steps of DeepFaceLab. We develop a lot of human-in-the-loop designes in the whole framework. For examples, DeepFaceLab can swap any area of faces and conduct high-quality face-swapping even under heavy occlusion with the help of few-shot human labeling and XSeg. There are also dozens of image editing algorithms available in the mergence phase to generate satisfactory results.•DeepFaceLab is the leading software for creating deepfakes. As an open-source project, DeepFaceLab has obtained more than 35,000 stars on Github. Face swapping has drawn a lot of attention for its compelling performance. However, current deepfake methods suffer the effects of obscure workflow and poor performance. To solve these problems, we present DeepFaceLab, the current dominant deepfake framework for practical face-swapping. It provides the necessary tools as well as an easy-to-use way to conduct high-quality face-swapping. It also offers a flexible and loose coupling structure for people who need to strengthen their pipeline with other features without writing complicated boilerplate code. We detail the principles that drive the implementation of DeepFaceLab and introduce its pipeline. DeepFaceLab could achieve cinema-level results with high fidelity as our supplemental video shows. We also demonstrate the advantage of our system by comparing our approach with other face-swapping methods. Deepfake defense not only requires the research of detection but also requires the efforts of generation methods. As for a popular and practical toolkit, we encourage users to promote harmless deepfake-entertainment content on social media, reminding the public of the existence of deepfake when they are looking for entertainment.