Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings

• Published in: IEEE transactions on software engineering Vol. 34; no. 4; pp. 485 - 496
• Main Authors: Lessmann, S., Baesens, B., Mues, C., Pietsch, S.
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.07.2008; IEEE Computer Society
• Subjects: Accuracy; Algorithms; Benchmark testing; Classification; Classifiers; Complexity measures; Computer based modeling; Computer programs; Convergence; Data mining; Datasets; Defects; Digital Object Identifier; Empirical analysis; Fault diagnosis; Formal methods; Hypotheses; Hypothesis testing; Large-scale systems; Mathematical models; NASA; Predictive models; Software; Software engineering; Software quality; Software systems; Software testing; Statistical analysis; Statistical methods; Test methods; Statistical methods; Data mining; Formal methods; Complexity measures
• ISSN: 0098-5589; 1939-3520
• DOI: 10.1109/TSE.2008.35

Software defect prediction strives to improve software quality and testing efficiency by constructing predictive classification models from code attributes to enable a timely identification of fault-prone modules. Several classification models have been evaluated for this task. However, due to inconsistent findings regarding the superiority of one classifier over another and the usefulness of metric-based classification in general, more research is needed to improve convergence across studies and further advance confidence in experimental results. We consider three potential sources for bias: comparing classifiers over one or a small number of proprietary data sets, relying on accuracy indicators that are conceptually inappropriate for software defect prediction and cross-study comparisons, and, finally, limited use of statistical testing procedures to secure empirical findings. To remedy these problems, a framework for comparative software defect prediction experiments is proposed and applied in a large-scale empirical comparison of 22 classifiers over 10 public domain data sets from the NASA Metrics Data repository. Overall, an appealing degree of predictive accuracy is observed, which supports the view that metric-based classification is useful. However, our results indicate that the importance of the particular classification algorithm may be less than previously assumed since no significant performance differences could be detected among the top 17 classifiers.