Practical Combinatorial Interaction Testing: Empirical Findings on Efficiency and Early Fault Detection

• Published in: IEEE transactions on software engineering Vol. 41; no. 9; pp. 901 - 924
• Main Authors: Petke, Justyna, Cohen, Myra B., Harman, Mark, Yoo, Shin
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.09.2015; IEEE Computer Society
• Subjects: Algorithms; Combinatorial analysis; Combinatorial Interaction Testing; Computer programs; Empirical Studies; Fault detection; Fault diagnosis; Faults; Flexible printed circuits; Genetic algorithms; Greedy algorithms; Handling; Prioritisation; Simulated annealing; Software engineering; Software Testing; Strength; Studies; Test systems; Testing; Turning; Combinatorial interaction testing; prioritisation; software testing; empirical studies
• ISSN: 0098-5589; 1939-3520
• DOI: 10.1109/TSE.2015.2421279

Combinatorial interaction testing (CIT) is important because it tests the interactions between the many features and parameters that make up the configuration space of software systems. Simulated Annealing (SA) and Greedy Algorithms have been widely used to find CIT test suites. From the literature, there is a widely-held belief that SA is slower, but produces more effective tests suites than Greedy and that SA cannot scale to higher strength coverage. We evaluated both algorithms on seven real-world subjects for the well-studied two-way up to the rarely-studied six-way interaction strengths. Our findings present evidence to challenge this current orthodoxy: real-world constraints allow SA to achieve higher strengths. Furthermore, there was no evidence that Greedy was less effective (in terms of time to fault revelation) compared to SA; the results for the greedy algorithm are actually slightly superior. However, the results are critically dependent on the approach adopted to constraint handling. Moreover, we have also evaluated a genetic algorithm for constrained CIT test suite generation. This is the first time strengths higher than 3 and constraint handling have been used to evaluate GA. Our results show that GA is competitive only for pairwise testing for subjects with a small number of constraints.