Addressing combinatorial experiments and scarcity of subjects by provably orthogonal and crossover experimental designs

• Published in: The Journal of systems and software Vol. 211; p. 111990
• Main Authors: Massacci, Fabio, Papotti, Aurora, Paramitha, Ranindya
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.05.2024
• Subjects: Crossover experimental design; Design of experiments; Full factorial design; Orthogonal design; Orthogonal design; Crossover experimental design; Full factorial design; Design of experiments
• ISSN: 0164-1212; 1873-1228
• DOI: 10.1016/j.jss.2024.111990

Experimentation in Software and Security Engineering is a common research practice, in particular with human subjects. The combinatorial nature of software configurations and the difficulty of recruiting experienced subjects or running complex and expensive experiments make the use of full factorial experiments unfeasible to obtain statistically significant results. Provide comprehensive alternative Designs of Experiments (DoE) based on orthogonal designs or crossover designs that provably meet desired requirements such as balanced pair-wise configurations or balanced ordering of scenarios to mitigate bias or learning effects. We also discuss and formalize the statistical implications of these design choices, in particular for crossover designs. We made available the algorithmic construction of the design for ℓ=2,3,4,5 levels for arbitrary K factors and illustrated their use with examples from security and software engineering research. •Algorithms to compute orthogonal designs (Taguchi designs).•Generation of balanced repeated crossover designs with different scenarios.•Formal proofs of design of experiments (DOE) properties and their statistical impact.•Examples of design of experiments in security and software engineering.•Artifacts in Excel and Jupyter Notebook for plug-and-play applicability.