Poster: A Weight-Based Approach to Combinatorial Test Generation

• Published in: 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion) pp. 378 - 383
• Main Authors: Zhao, Jing, Ning, Gaorong, Lu, Hualin, Wang, Yanbin, Yan, Cai, Zhang, Jian
• Format: Conference Proceeding
• Language: English
• Published: ACM 01.05.2018
• Subjects: Benchmark testing; combinatorial testing; Fault detection; fault detection rate; Measurement; Software; Software engineering; Software testing; test case selection; weighted combinatorial coverage
• ISSN: 2574-1934

Combinatorial testing (CT) is very efficient to test parameterized systems. Kuhn et al. investigated the interaction faults of some real programs, and found that the faulty combinations are caused by the combination of no more than 6 parameters. Three or fewer parameters triggered a total of almost 90% of the failures in the application[3]. However, for high-quality software, simply testing all 3-way combinations is not sufficient [5], which may increase the risk of residual errors that lead to system failures and security weakness[4]. In addition, the number of test cases at 100% coverage for high-way is huge, which is beyond the farthest test overhead restrictions. Covering array is typically used as the test suite in CT, which should convey much information for the fault detection. We firstly proposed a weighted combinatorial coverage (CC), focusing on the fault detection capability of each test case instead of 100% percent t-way CC. Secondly, we give the test case selection algorithm FWA (fixed weight algorithm) using weighted CC metric. For generating each test case, our method first randomly generates several candidates, and selects the one that has the highest fault detection possibility with the different sampling pool size. Thirdly, we give the theorems for our algorithm and definitions for the weighted CC. Finally, we compared the selected sample sized and the fault-detection capabilities of FWA as well as t-wise algorithms by using the four benchmarks with configuration options interaction faults, and we found FWA is able to detect higher number of faults with the less selected sample size, specifically, FWA is able to detect high-wise interaction faults with the less selected sample size compared with the 4-wise as well as 5-wise algorithms.