Face Relighting from a Single Image under Arbitrary Unknown Lighting Conditions

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 31; no. 11; pp. 1968 - 1984
• Main Authors: Yang Wang, Lei Zhang, Zicheng Liu, Gang Hua, Zhen Wen, Zhengyou Zhang, Samaras, D.
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01.11.2009; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D spherical harmonic basis morphable model; Albedo; Algorithms; Applied sciences; Approximation error; Artificial Intelligence; Biometry - methods; Computer graphics; Computer science; control theory; systems; Exact sciences and technology; Face - anatomy & histology; Face detection; Face recognition; Face synthesis and recognition; Humans; Illumination; Image databases; Image Enhancement - methods; Image generation; Image Interpretation, Computer-Assisted - methods; Lighting; Lighting - methods; Markov random field; Markov random fields; Mathematical models; Pattern Recognition, Automated - methods; Pattern recognition. Digital image processing. Computational geometry; Photography - methods; Reproducibility of Results; Sensitivity and Specificity; Shape; Spherical harmonics; Subtraction Technique; Surface layer; Texture; vision for graphics; Modeling and recovery of physical attributes; Face and gesture recognition; Markov process; Statistical analysis; Spherical shape; Probabilistic approach; Face recognition; Image processing; 3D spherical harmonic basis morphable model; Worst case method; Markov model; Pattern recognition; Markov random field; Modeling; Texture; Graphics; Luminance; Albedo; Vector space; Lighting; Face synthesis and recognition; Facies; Illumination; Approximation error; Spherical harmonic; vision for graphics
• ISSN: 0162-8828; 1939-3539; 1939-3539
• DOI: 10.1109/TPAMI.2008.244

In this paper, we present a new method to modify the appearance of a face image by manipulating the illumination condition, when the face geometry and albedo information is unknown. This problem is particularly difficult when there is only a single image of the subject available. Recent research demonstrates that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace using a spherical harmonic representation. Moreover, morphable models are statistical ensembles of facial properties such as shape and texture. In this paper, we integrate spherical harmonics into the morphable model framework by proposing a 3D spherical harmonic basis morphable model (SHBMM). The proposed method can represent a face under arbitrary unknown lighting and pose simply by three low-dimensional vectors, i.e., shape parameters, spherical harmonic basis parameters, and illumination coefficients, which are called the SHBMM parameters. However, when the image was taken under an extreme lighting condition, the approximation error can be large, thus making it difficult to recover albedo information. In order to address this problem, we propose a subregion-based framework that uses a Markov random field to model the statistical distribution and spatial coherence of face texture, which makes our approach not only robust to extreme lighting conditions, but also insensitive to partial occlusions. The performance of our framework is demonstrated through various experimental results, including the improved rates for face recognition under extreme lighting conditions.