Data-driven Evaluation of Visual Quality Measures

• Published in: Computer graphics forum Vol. 34; no. 3; pp. 201 - 210
• Main Authors: Sedlmair, M., Aupetit, M.
• Format: Journal Article
• Language: English
• Published: Oxford Blackwell Publishing Ltd 01.06.2015
• Subjects: Algorithms; Analysis; Categories and Subject Descriptors (according to ACM CCS); Ground truth; H.5.0 [Information Interfaces and Presentation]: General; Human; Image processing systems; Judgment; Judgments; Pattern recognition systems; Separation; State of the art; Studies; Two dimensional; Visual
• ISSN: 0167-7055; 1467-8659
• DOI: 10.1111/cgf.12632

Visual quality measures seek to algorithmically imitate human judgments of patterns such as class separability, correlation, or outliers. In this paper, we propose a novel data‐driven framework for evaluating such measures. The basic idea is to take a large set of visually encoded data, such as scatterplots, with reliable human “ground truth” judgements, and to use this human‐labeled data to learn how well a measure would predict human judgements on previously unseen data. Measures can then be evaluated based on predictive performance—an approach that is crucial for generalizing across datasets but has gained little attention so far. To illustrate our framework, we use it to evaluate 15 state‐of‐the‐art class separation measures, using human ground truth data from 828 class separation judgments on color‐coded 2D scatterplots.