Joint learning of visual and spatial features for edit propagation from a single image

• Published in: The Visual computer Vol. 36; no. 3; pp. 469 - 482
• Main Authors: Gui, Yan, Zeng, Guang
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.03.2020; Springer Nature B.V
• Subjects: Approximation; Artificial Intelligence; Artificial neural networks; Classification; Colorization; Computer Graphics; Computer Science; Conditional random fields; Deep learning; Editing; Image Processing and Computer Vision; Machine learning; Neural networks; Optimization techniques; Original Article; Propagation; Semantics; Image editing; Deep neural network; Edit propagation; Fully connected conditional random field
• ISSN: 0178-2789; 1432-2315
• DOI: 10.1007/s00371-019-01633-6

In this paper, we regard edit propagation as a multi-class classification problem and deep neural network (DNN) is used to solve the problem. We design a shallow and fully convolutional DNN that can be trained end-to-end. To achieve this, our method uses combinations of low-level visual features, which are extracted from the input image, and spatial features, which are computed through transforming user interactions, as input of the DNN, which efficiently performs a joint learning of visual and spatial features. We then train the DNN on many of such combinations in order to build a DNN-based pixel-level classifier. Our DNN is also equipped with patch-by-patch training and whole image estimation, speeding up learning and inference. Finally, we improve classification accuracy of the DNN by employing a fully connected conditional random field. Experimental results show that our method can respond to user interactions well and generate precise results compared with the state-of-art edit propagation approaches. Furthermore, we demonstrate our method on various applications.