Crossover Designs in Software Engineering Experiments: Benefits and Perils

In experiments with crossover design subjects apply more than one treatment. Crossover designs are widespread in software engineering experimentation: they require fewer subjects and control the variability among subjects. However, some researchers disapprove of crossover designs. The main criticism...

Full description

Saved in:
Bibliographic Details
Published inIEEE transactions on software engineering Vol. 42; no. 2; pp. 120 - 135
Main Authors Vegas, Sira, Apa, Cecilia, Juristo, Natalia
Format Journal Article
LanguageEnglish
Published New York IEEE 01.02.2016
IEEE Computer Society
Subjects
Animals
Atmospheric measurements
Avoidance
carryover
Constraining
crossover designs
Crossovers
Data analysis
Design
Design analysis
Design engineering
Experiments
Information processing
Literature reviews
Particle measurements
Psychology
Risk assessment
Software engineering
Studies
Threat evaluation
US Government agencies
controlled experiment
carryover
data analysis
Experimental software engineering
crossover design
Online AccessGet full text

Cover

More Information
Summary:In experiments with crossover design subjects apply more than one treatment. Crossover designs are widespread in software engineering experimentation: they require fewer subjects and control the variability among subjects. However, some researchers disapprove of crossover designs. The main criticisms are: the carryover threat and its troublesome analysis. Carryover is the persistence of the effect of one treatment when another treatment is applied later. It may invalidate the results of an experiment. Additionally, crossover designs are often not properly designed and/or analysed, limiting the validity of the results. In this paper, we aim to make SE researchers aware of the perils of crossover experiments and provide risk avoidance good practices. We study how another discipline (medicine) runs crossover experiments. We review the SE literature and discuss which good practices tend not to be adhered to, giving advice on how they should be applied in SE experiments. We illustrate the concepts discussed analysing a crossover experiment that we have run. We conclude that crossover experiments can yield valid results, provided they are properly designed and analysed, and that, if correctly addressed, carryover is no worse than other validity threats.
Bibliography:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-1
ObjectType-Feature-2
content type line 23
ISSN:0098-5589
1939-3520
DOI:10.1109/TSE.2015.2467378