OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples

• Published in: IEEE transactions on visualization and computer graphics Vol. 27; no. 7; pp. 3335 - 3349
• Main Authors: Chen, Changjian, Yuan, Jun, Lu, Yafeng, Liu, Yang, Su, Hang, Yuan, Songtao, Liu, Shixia
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.07.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Approximation algorithms; Cats; Complexity; Context; Dogs; Feature extraction; grid layout; interactive visualization; Layout; Layouts; Out-of-Distribution detection; Performance degradation; Performance prediction; Prediction models; Training; Visualization
• ISSN: 1077-2626; 1941-0506; 1941-0506
• DOI: 10.1109/TVCG.2020.2973258

One major cause of performance degradation in predictive models is that the test samples are not well covered by the training data. Such not well-represented samples are called OoD samples. In this article, we propose OoDAnalyzer, a visual analysis approach for interactively identifying OoD samples and explaining them in context. Our approach integrates an ensemble OoD detection method and a grid-based visualization. The detection method is improved from deep ensembles by combining more features with algorithms in the same family. To better analyze and understand the OoD samples in context, we have developed a novel <inline-formula><tex-math notation="LaTeX">k</tex-math> <mml:math><mml:mi>k</mml:mi></mml:math><inline-graphic xlink:href="liu-ieq1-2973258.gif"/> </inline-formula>NN-based grid layout algorithm motivated by Hall's theorem. The algorithm approximates the optimal layout and has <inline-formula><tex-math notation="LaTeX">O(kN^2)</tex-math> <mml:math><mml:mrow><mml:mi>O</mml:mi><mml:mo>(</mml:mo><mml:mi>k</mml:mi><mml:msup><mml:mi>N</mml:mi><mml:mn>2</mml:mn></mml:msup><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="liu-ieq2-2973258.gif"/> </inline-formula> time complexity, faster than the grid layout algorithm with overall best performance but <inline-formula><tex-math notation="LaTeX">O(N^3)</tex-math> <mml:math><mml:mrow><mml:mi>O</mml:mi><mml:mo>(</mml:mo><mml:msup><mml:mi>N</mml:mi><mml:mn>3</mml:mn></mml:msup><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="liu-ieq3-2973258.gif"/> </inline-formula> time complexity. Quantitative evaluation and case studies were performed on several datasets to demonstrate the effectiveness and usefulness of OoDAnalyzer.