What Not to Test (for Cyber-Physical Systems)

• Published in: IEEE transactions on software engineering Vol. 49; no. 7; pp. 1 - 17
• Main Authors: Ling, Xiao, Menzies, Tim
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.07.2023; IEEE Computer Society
• Subjects: Approximation; Cyber-physical systems; History; Mathematical analysis; Modeling and Model-Driven Engineering; Multi-goal Optimization; Mutation; Optimization; Regression tree analysis; Search-based Software Engineering; Simulation-based Testing; Software testing; Task analysis; Testing; Validation and Verification
• ISSN: 0098-5589; 1939-3520
• DOI: 10.1109/TSE.2023.3272309

For simulation-based systems, finding a set of test cases with the least cost by exploring multiple goals is a complex task. Domain-specific optimization goals (e.g. maximize output variance) are useful for guiding the rapid selection of test cases via mutation. But evaluating the selected test cases via mutation (that can distinguish the current program from the mutated systems) is a different goal to domain-specific optimizations. While the optimization goals can be used to guide the mutation analysis, that guidance should be viewed as a weak indicator since it can hurt the mutation effectiveness goals by focusing too much on the optimization goals. Based on the above, this paper proposes DoLesS ( Do mination with Le a s t S quares Approximation) that selects the minimal and effective test cases by averaging over a coarse-grained grid of the information gained from multiple optimizations goals. DoLesS applies an inverted least squares approximation approach to find a minimal set of tests that can distinguish better from worse parts of the optimization goals. When tested on multiple simulation-based systems, DoLesS performs as well or even better as the prior state-of-the-art, while running 80-360 times faster on average (seconds instead of hours). For replication purposes, all our code is on-line: https://github.com/ai-se/DoLesS .