Continuous experimentation and A/B testing a mapping study

• Published in: 2018 IEEE ACM 4th International Workshop on Rapid Continuous Software Engineering (RCoSE) pp. 35 - 41
• Main Authors: Ros, Rasmus, Runeson, Per
• Format: Conference Proceeding
• Language: English
• Published: New York, NY, USA ACM 29.05.2018
• Series: ACM Conferences
• Subjects: A/B testing; Companies; Continuous experimentation; Data mining; General and reference > Document types > Surveys and overviews; Industries; Mapping study; Software; Software engineering; Testing; mapping study; A/B testing; continuous experimentation
• ISBN: 9781450357456; 1450357458
• DOI: 10.1145/3194760.3194766

Background. Continuous experimentation (CE) has recently emerged as an established industry practice and as a research subject. Our aim is to study the application of CE and A/B testing in various industrial contexts. Objective. We wanted to investigate whether CE is used in different sectors of industry, by how it is reported in academic studies. We also wanted to explore the main topics researched to give an overview of the subject and discuss future research directions. Method. We performed a systematic mapping study of the published literature and included 62 papers, using a combination of database search and snowballing. Results. Most reported software experiments are done online and with software delivered as a service, although varied exemptions exist for e.g., financial software and games. The most frequently researched topics are challenges to conduct experiments and statistical methods for software experiments. Conclusions. The software engineering research on CE is still in its infancy. There are future research opportunities in evaluation research of technical topics and investigations of ethical experimentation. We conclude that the included studies show that A/B testing is applicable to a diversity of software and organisations.