Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures

• Published in: Information and software technology Vol. 62; pp. 198 - 213
• Main Authors: Nie, Changhai, Wu, Huayao, Niu, Xintao, Kuo, Fei-Ching, Leung, Hareton, Colbourn, Charles J.
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01.06.2015; Elsevier Science Ltd
• Subjects: Adaptive Random Testing (ART); Business metrics; Combinatorial Testing (CT); Comparative analysis; Failure analysis; Interaction Triggered Failure (ITF); Minimal Failure-causing Schema (MFS); Random Testing (RT); Software engineering; Software testing; Studies; Test systems; Software testing; Random Testing (RT); Combinatorial Testing (CT); Interaction Triggered Failure (ITF); Minimal Failure-causing Schema (MFS); Adaptive Random Testing (ART)
• ISSN: 0950-5849; 1873-6025
• DOI: 10.1016/j.infsof.2015.02.008

Although ART can greatly enhance RT and t-way CT is better than ART and RT, normally in hitting t′-MFS:(1)There are only from 10% to 30% difference when t=t′, see Figs. 6, 16, and 26.(2)The difference become quite smaller when t′<t, see Figs. 15, 24, and 25.(3)When t′>t, it may not be better, see Figs. 7, 8, and 17 .So we can understand why some researchers claim that there is no significant difference between RT and CT. (See the results from the above figures, where Sn(t)(t′-cov), t, t′=2, 3, or 4 represents the t′-way combination coverage of the same size test suite of the t-way covering array generated by different methods, more details can be found from paper.) [Display omitted] •We make a systematic comparison of CT, RT and ART under the scenario of hitting MFS.•We conducted two kinds of experiments to compare them with four metrics.•Normally in hitting t′-MFS there are only from 10% to 30% difference when t=t′.•ART can greatly enhance RT, but it depends on the implementation. Software behavior depends on many factors, and some failures occur only when certain factors interact. This is known as an interaction triggered failure, and the corresponding selection of factor values can be modeled as a Minimal Failure-causing Schema (MFS). (An MFS involving m factors is an m-MFS.) Combinatorial Testing (CT) has been developed to exercise (“hit”) all MFS with few tests. Adaptive Random Resting (ART) endeavors to make tests as different as possible, ensuring that testing of MFS is not unnecessarily repeated. Random Testing (RT) chooses tests at random without regard to the MFS already treated. CT might be expected to improve on RT for finding interaction triggered faults, and yet some studies report no significant difference. CT can also be expected to be better than ART, and yet other studies report that ART can be much better than RT. In light of these, the relative merits of CT, ART, and RT for finding interaction triggered faults are unclear. To investigate the relationships among CT, ART, and RT, we conduct the first complete and systematic comparison for the purpose of hitting MFS. A systematic review of six aspects of CT, RT and ART is conducted first. Then two kinds of experiments are used to compare them under four metrics. ART improves upon RT, but t-way CT is better than both. In hitting t′-MFS the advantage is typically in the range from 10% to 30% when t=t′, but becomes much smaller when t′<t, and there may be no advantage when t′>t. The latter case may explain the studies reporting no significant difference between RT and CT. RT is easily implemented. However, depending on its implementation, ART can improve upon RT. CT does as well as ART whether or nott′=t, but provides a valuable improvement in the cases when t′=t.