METTLE: A METamorphic Testing Approach to Assessing and Validating Unsupervised Machine Learning Systems

• Published in: IEEE transactions on reliability Vol. 69; no. 4; pp. 1293 - 1322
• Main Authors: Xie, Xiaoyuan, Zhang, Zhiyi, Chen, Tsong Yueh, Liu, Yang, Poon, Pak-Lok, Xu, Baowen
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adequacy; Artificial intelligence; Benchmark testing; Clustering; Clustering assessment; clustering validation; End users; end-user software engineering; Evaluation; Machine learning; metamorphic relation (MR); metamorphic testing (MT); Software engineering; Unsupervised learning; unsupervised machine learning; User requirements
• ISSN: 0018-9529; 1558-1721
• DOI: 10.1109/TR.2020.2972266

Unsupervised machine learning is the training of an artificial intelligence system using information that is neither classified nor labeled, with a view to modeling the underlying structure or distribution in a dataset. Since unsupervised machine learning systems are widely used in many real-world applications, assessing the appropriateness of these systems and validating their implementations with respect to individual users' requirements and specific application scenarios/contexts are indisputably two important tasks. Such assessments and validation tasks, however, are fairly challenging due to the absence of a priori knowledge of the data. In view of this challenge, in this article, we develop a MET amorphic T esting approach to assessing and validating unsupervised machine LE arning systems, abbreviated as mettle . Our approach provides a new way to unveil the (possibly latent) characteristics of various machine learning systems, by explicitly considering the specific expectations and requirements of these systems from individual users' perspectives. To support mettle , we have further formulated 11 generic metamorphic relations (MRs), covering users' generally expected characteristics that should be possessed by machine learning systems. We have performed an experiment and a user evaluation study to evaluate the viability and effectiveness of mettle . Our experiment and user evaluation study have shown that, guided by user-defined MR-based adequacy criteria, end users are able to assess, validate, and select appropriate clustering systems in accordance with their own specific needs. Our investigation has also yielded insightful understanding and interpretation of the behavior of the machine learning systems from an end-user software engineering's perspective , rather than a designer's or implementor's perspective, who normally adopts a theoretical approach.