Exemplar-Based Image and Video Stylization Using Fully Convolutional Semantic Features

• Published in: IEEE transactions on image processing Vol. 26; no. 7; pp. 3542 - 3555
• Main Authors: Zhu, Feida, Yan, Zhicheng, Bu, Jiajun, Yu, Yizhou
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.07.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Automation; Clips; Color; color transform; Computer architecture; Datasets; Feature extraction; fully convolutional networks; Image color analysis; Image enhancement; Image stylization; Learning; Machine learning; Neural networks; Semantics; Social networks; Transforms
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2017.2703099

Color and tone stylization in images and videos strives to enhance unique themes with artistic color and tone adjustments. It has a broad range of applications from professional image postprocessing to photo sharing over social networks. Mainstream photo enhancement softwares, such as Adobe Lightroom and Instagram, provide users with predefined styles, which are often hand-crafted through a trial-and-error process. Such photo adjustment tools lack a semantic understanding of image contents and the resulting global color transform limits the range of artistic styles it can represent. On the other hand, stylistic enhancement needs to apply distinct adjustments to various semantic regions. Such an ability enables a broader range of visual styles. In this paper, we first propose a novel deep learning architecture for exemplar-based image stylization, which learns local enhancement styles from image pairs. Our deep learning architecture consists of fully convolutional networks for automatic semantics-aware feature extraction and fully connected neural layers for adjustment prediction. Image stylization can be efficiently accomplished with a single forward pass through our deep network. To extend our deep network from image stylization to video stylization, we exploit temporal superpixels to facilitate the transfer of artistic styles from image exemplars to videos. Experiments on a number of data sets for image stylization as well as a diverse set of video clips demonstrate the effectiveness of our deep learning architecture.