Selfie retoucher: subject-oriented self-portrait enhancement

• Published in: Multimedia tools and applications Vol. 78; no. 19; pp. 27591 - 27609
• Main Authors: Xia, Sifeng, Yang, Shuai, Liu, Jiaying
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.10.2019; Springer Nature B.V
• Subjects: Algorithms; Cameras; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Field of view; Fields (mathematics); Illumination; Lager; Lighting; Markov processes; Methods; Multimedia; Multimedia Information Systems; Optimization; Photographers; Portable equipment; Registration; Smartphones; Social networks; Special Purpose and Application-Based Systems; Image registration; Field of view expansion; Image enhancement; Illumination unification
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-019-07873-x

Sharing self-portraits starts trending nowadays with the boom of social networks and the rise of smartphones. However, limited by the hardware capabilities, self-portraits taken by the front cameras of portable media devices usually face quality problems such as an incomplete field of view and poor lighting style. In our paper, we introduce a selfie retoucher which enhances a self-portrait with the help of N supporting photos that share the same scene and similar shooting time. With the extra information brought by the supporting photos, a lager field of view and a better lighting style can be achieved. To accomplish this, we propose a novel subject-oriented self-portrait enhancement method with a cascaded illumination unification and photos registration framework. Based on the correspondences extracted from the input 1+ N photos, our method estimates and updates the illumination and registration coefficients in a cascaded manner. Moreover, a subject-oriented enhancement algorithm is proposed to enhance the face of the photographer in the self-portrait. We adopt a face-specific illumination correction process over the self-portrait to further improve the visual quality of the subject. After the enhancement, we globally fuse the aligned photos by a Markov Random Field based optimization method. During the fusion, a body map is additionally derived from the subject for guidance. Experimental results demonstrate that the proposed method achieves high-quality results in this novel application scenario.