Characterizing Flaky Tests in Node.js Applications

Regression testing is an important means of assessing the quality of Node.js applications. However, non-deterministic executions inside Node.js framework could make test cases intermittently pass or fail on the same version of code, which are called flaky tests. Flaky tests can cause unreliable test...

Full description

Saved in:
Bibliographic Details
Published in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) pp. 1753 - 1755
Main Authors Chang, Xiaoning, Liang, Zheheng, Wu, Guoquan, Gao, Yu, Chen, Wei, Wei, Jun, Long, Zhenyue, Cui, Lei, Huang, Tao
Format Conference Proceeding
LanguageEnglish
Published IEEE 11.09.2023
Subjects
Codes
Computer bugs
Debugging
flaky tests
non-determinism
Software engineering
Task analysis
Testing
Online AccessGet full text

Cover

More Information
Summary:Regression testing is an important means of assessing the quality of Node.js applications. However, non-deterministic executions inside Node.js framework could make test cases intermittently pass or fail on the same version of code, which are called flaky tests. Flaky tests can cause unreliable test results, and make developers waste a significant amount of time debugging the bugs that do not belong to the target application. In this paper, we conduct an empirical study on 87 flaky tests from 7 popular Node.js applications, and analyze the non-determinism that causes these flaky tests. Through this study, there is a wide range of non-determinism to cause flaky tests, including non-deterministic event triggering order, non-deterministic function calls, non-deterministic process/thread scheduling order, non-deterministic execution of asynchronous tasks and non-deterministic event triggering data. The result reveals that, existing approaches on event race detection are not sufficient for flaky test detection. In future, researchers can design flaky test detection approaches targeted at different categories of non-determinism.
ISSN:2643-1572
DOI:10.1109/ASE56229.2023.00025