Face Illumination Transfer and Swapping via Dense Landmark and Semantic Parsing

• Published in: IEEE sensors journal Vol. 22; no. 18; pp. 17391 - 17398
• Main Authors: Jin, Xin, Li, Zhonglan, Ning, Ning, Lu, Huimin, Li, Xiaodong, Zhang, Xiaokun, Zhu, Xingfan, Fang, Xi
• Format: Journal Article
• Language: English
• Published: New York IEEE 15.09.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D reconstruction; Computer vision; Cross cutting; Face image; Face recognition; Faces; Illumination; illumination swap; illumination transfer; Image reconstruction; Image transmission; Light; Lighting; Rendering; semantic parsing; Semantics; Sensors; Solid modeling; Strain; Three dimensional analysis; Three dimensional models; Three-dimensional displays; Virtual reality
• ISSN: 1530-437X; 1558-1748
• DOI: 10.1109/JSEN.2020.3025918

The image-based virtual illumination technology directly changes the illumination effect of objects on the image. It does not require complex light propagation simulation calculations, it is an image-based rendering technology. The image mainly relies on the imaging of the visual sensor, and uses the virtual illumination technology of the image to illuminate the image obtained by the visual sensor. This is a new cross-cutting direction in computer vision, virtual reality and other fields. Face Illumination Swapping via Dense Landmark and Semantic Parsing is a major branch. Keeping the geometrical features of the target images and relighting the entire image instead of the face area are problems to be solved in the research. This paper based on the three-dimensional model to analyze the illumination information of the face images and re-render the illumination of the target face, and finally achieves the illumination swap between the two face images. We designed and implemented a 3DDFA-based face image illumination transfer method. First, 3DDFA is used to reconstruct the target face image. Estimate the surface normal and albedo. Then align and fill the surface normal and face parsing to illuminate the face image for light rendering and illumination transfer of the face images. Finally, the illumination analysis and re-rendering of face images based on 3DDFA are expanded to achieve the swap of illumination between face images. Experimental results show that this method can generate good effects of face image illumination transmission and swap while keeping the geometric features of the target images.