CAST: Learning Both Geometric and Texture Style Transfers for Effective Caricature Generation

• Published in: IEEE transactions on image processing Vol. 31; pp. 3347 - 3358
• Main Authors: Huo, Jing, Liu, Xiangde, Li, Wenbin, Gao, Yang, Yin, Hujun, Luo, Jiebo
• Format: Journal Article
• Language: English
• Published: United States IEEE 2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: caricature generation; Covariance matrices; Face recognition; Faces; Generative adversarial networks; Image processing; Modules; Object recognition; semantic alignment; Semantics; Shape; Style transfer; Task analysis; Texture
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2022.3154238

Given a photo of a subject, ability to generate a caricature image that captures distinct characteristics of the subject but with certain exaggeration of their prominent features is of fundamental importance to image processing and facial recognition. There are two main challenges in this task: shape exaggeration and style transfer. The former morphs and exaggerates key facial features of the subject, while the latter generates caricature images in a certain artistic style. In this paper, we propose a CAricature Style Transfer (CAST) framework for caricature generation. There are two modules in the proposed framework. The first is a geometric warping module. Different from the existing style transfer methods, we incorporate the Whitening and Coloring Transformation (WCT) in the geometric style transfer. The WCT is learned on photo and caricature landmarks or the caricature landmark space of a specific artist and is capable of transforming input photo landmarks to caricature landmarks. The second module is a texture style rendering module. We propose a new style transfer method by considering a semantic region-aligned style transfer via affinity constraint. Given a reference caricature image as the style reference, this module is capable of transferring styles between the same or similar semantic regions in caricatures and photos. Furthermore, it can transfer visual attributes of the reference caricatures (such as mouth shape and expressions) to the output caricatures. Experiments have shown desirable effects of the proposed method in transferring both the geometric and artistic texture styles of caricatures. Both qualitative and quantitative results show that the CAST framework is more effective compared than the state-of-the-art caricature generation methods.