Discover, Explain, Improve: An Automatic Slice Detection Benchmark for Natural Language Processing

• Published in: Transactions of the Association for Computational Linguistics Vol. 11; pp. 1537 - 1552
• Main Authors: Hua, Wenyue, Jin, Lifeng, Song, Linfeng, Mi, Haitao, Zhang, Yongfeng, Yu, Dong
• Format: Journal Article
• Language: English
• Published: Cambridge MIT Press Journals, The 14.12.2023
• Subjects: Benchmarks; Computer vision; Error analysis; Error detection; Machine learning; Natural language processing; Quantitative analysis; Semantic features; Systematic errors
• ISSN: 2307-387X; 2307-387X
• DOI: 10.1162/tacl_a_00617

Abstract Pretrained natural language processing (NLP) models have achieved high overall performance, but they still make systematic errors. Instead of manual error analysis, research on slice detection models (SDMs), which automatically identify underperforming groups of datapoints, has caught escalated attention in Computer Vision for both understanding model behaviors and providing insights for future model training and designing. However, little research on SDMs and quantitative evaluation of their effectiveness have been conducted on NLP tasks. Our paper fills the gap by proposing a benchmark named “Discover, Explain, Improve (DEIm)” for classification NLP tasks along with a new SDM Edisa. Edisa discovers coherent and underperforming groups of datapoints; DEIm then unites them under human-understandable concepts and provides comprehensive evaluation tasks and corresponding quantitative metrics. The evaluation in DEIm shows that Edisa can accurately select error-prone datapoints with informative semantic features that summarize error patterns. Detecting difficult datapoints directly boosts model performance without tuning any original model parameters, showing that discovered slices are actionable for users.1