Developing and applying a benchmark for evaluating image stylization

• Published in: Computers & graphics Vol. 67; pp. 58 - 76
• Main Authors: Mould, David, Rosin, Paul L.
• Format: Journal Article
• Language: English
• Published: Oxford Elsevier Ltd 01.10.2017; Elsevier Science Ltd
• Subjects: Abstraction; Algorithms; Benchmarks; Computer graphics; Computer vision; Evaluation; Image processing; Image processing systems; Image stylization; Non-photorealistic rendering; Non-photorealistic rendering; Benchmarks; Evaluation; Image processing; Image stylization; Abstraction
• ISSN: 0097-8493; 1873-7684
• DOI: 10.1016/j.cag.2017.05.025

•Proposing a more systematic approach to image stylization evaluation, with a common set of images.•Presenting a 20-image benchmark for helping NPR practitioners evaluate their methods.•Demonstrating the use of the benchmark set with six existing image stylization methods. [Display omitted] The non-photorealistic rendering community has had difficulty evaluating its research results. Other areas of computer graphics, and related disciplines such as computer vision, have made progress by comparing algorithms’ performance on common datasets, or benchmarks. We argue for the benefits of establishing a benchmark image set to which image stylization methods can be applied, simplifying the comparison of methods, and broadening the testing to which a given method is subjected. We propose a set of benchmark images, representing a range of possible subject matter and image features of interest to researchers, and we describe the policies, tradeoffs, and reasoning that led us to the particular images in the set. Then, we apply six previously existing stylization algorithms to the benchmark images; we discuss observations arising from the interactions between the algorithms and the benchmark images. Inasmuch as the benchmark images were able to thoroughly exercise the algorithms and produce new material for discussion, we can conclude that the benchmark will be effective for its stated aim.