In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation

• Published in: IEEE transactions on visualization and computer graphics Vol. 25; no. 1; pp. 903 - 913
• Main Authors: Hullman, Jessica, Xiaoli Qiao, Correll, Michael, Kale, Alex, Kay, Matthew
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Confidence; Data visualization; Decision making; Decisions; Human behavior; Judgments; Measurement uncertainty; probability distribution; Semantics; subjective confidence; Task analysis; Taxonomy; Uncertainty; Uncertainty visualization; user study; Visualization
• ISSN: 1077-2626; 1941-0506; 1941-0506
• DOI: 10.1109/TVCG.2018.2864889

Understanding and accounting for uncertainty is critical to effectively reasoning about visualized data. However, evaluating the impact of an uncertainty visualization is complex due to the difficulties that people have interpreting uncertainty and the challenge of defining correct behavior with uncertainty information. Currently, evaluators of uncertainty visualization must rely on general purpose visualization evaluation frameworks which can be ill-equipped to provide guidance with the unique difficulties of assessing judgments under uncertainty. To help evaluators navigate these complexities, we present a taxonomy for characterizing decisions made in designing an evaluation of an uncertainty visualization. Our taxonomy differentiates six levels of decisions that comprise an uncertainty visualization evaluation: the behavioral targets of the study, expected effects from an uncertainty visualization, evaluation goals, measures, elicitation techniques, and analysis approaches. Applying our taxonomy to 86 user studies of uncertainty visualizations, we find that existing evaluation practice, particularly in visualization research, focuses on Performance and Satisfaction-based measures that assume more predictable and statistically-driven judgment behavior than is suggested by research on human judgment and decision making. We reflect on common themes in evaluation practice concerning the interpretation and semantics of uncertainty, the use of confidence reporting, and a bias toward evaluating performance as accuracy rather than decision quality. We conclude with a concrete set of recommendations for evaluators designed to reduce the mismatch between the conceptualization of uncertainty in visualization versus other fields.