Interactive Sketch-Driven Image Synthesis

• Published in: Computer graphics forum Vol. 34; no. 8; pp. 130 - 142
• Main Authors: Turmukhambetov, Daniyar, Campbell, Neill D.F., Goldman, Dan B, Kautz, Jan
• Format: Journal Article
• Language: English
• Published: Oxford Blackwell Publishing Ltd 01.12.2015
• Subjects: Analysis; Collection; Composing; Computer graphics; I.3.6 [Computer Graphics]: Interaction techniques Applications; Image processing systems; image synthesis; image-based rendering; Inspiration; Interactive; Interactive computer systems; Sketches; sketching; Studies; Visual; Workflow
• ISSN: 0167-7055; 1467-8659
• DOI: 10.1111/cgf.12665

We present an interactive system for composing realistic images of an object under arbitrary pose and appearance specified by sketching. Our system draws inspiration from a traditional illustration workflow: The user first sketches rough ‘masses’ of the object, as ellipses, to define an initial pose that can then be refined with more detailed contours as desired. The system is made robust to partial or inaccurate sketches using a reduced‐dimensionality model of pose space learnt from a labelled collection of photos. Throughout the composition process, interactive visual feedback is provided to guide the user. Finally, the user's partial or complete sketch, complemented with appearance requirements, is used to constrain the automatic synthesis of a novel, high‐quality, realistic image. We present an interactive system for composing realistic images of an object under arbitrary pose and appearance specified by sketching. Our system draws inspiration from a traditional illustration workflow: The user first sketches rough ‘masses’ of the object, as ellipses, to define an initial pose that can then be refined with more detailed contours as desired. The system is made robust to partial or inaccurate sketches using a reduced‐dimensionality model of pose space learnt from a labelled collection of photos. Throughout the composition process, interactive visual feedback is provided to guide the user.