Software test results exploration and visualization with continuous integration and nightly testing

Software testing is key for quality assurance of embedded systems. However, with increased development pace, the amount of test results data risks growing to a level where exploration and visualization of the results are unmanageable. This paper covers a tool, Tim, implemented at a company developin...

Full description

Saved in:
Bibliographic Details
Published inInternational journal on software tools for technology transfer Vol. 24; no. 2; pp. 261 - 285
Main Authors Strandberg, Per Erik, Afzal, Wasif, Sundmark, Daniel
Format Journal Article
LanguageEnglish
Published Berlin/Heidelberg Springer Berlin Heidelberg 01.04.2022
Springer Nature B.V
Subjects
Anomalies
Computer Science
Embedded systems
Exploration
Meetings
Quality assurance
Risk management
Software
Software development
Software Engineering
Software Engineering/Programming and Operating Systems
Software testing
Source code
Test results
Theory of Computation
Visual analytics
Visualization
Software testing
Visual analytics
Test results
Online AccessGet full text

Cover

More Information
Summary:Software testing is key for quality assurance of embedded systems. However, with increased development pace, the amount of test results data risks growing to a level where exploration and visualization of the results are unmanageable. This paper covers a tool, Tim, implemented at a company developing embedded systems, where software development occurs in parallel branches and nightly testing is partitioned over software branches, test systems and test cases. Tim aims to replace a previous solution with problems of scalability, requirements and technological flora. Tim was implemented with a reference group over several months. For validation, data were collected both from reference group meetings and logs from the usage of the tool. Data were analyzed quantitatively and qualitatively. The main contributions from the study include the implementation of eight views for test results exploration and visualization, the identification of four solutions patterns for these views (filtering, aggregation, previews and comparisons), as well as six challenges frequently discussed at reference group meetings (expectations, anomalies, navigation, integrations, hardware details and plots). Results are put in perspective with related work and future work is proposed, e.g., enhanced anomaly detection and integrations with more systems such as risk management, source code and requirements repositories.
ISSN:1433-2779
1433-2787
1433-2787
DOI:10.1007/s10009-022-00647-1