Seeding and Mocking in White-Box Fuzzing Enterprise RPC APIs: An Industrial Case Study

• Published in: IEEE/ACM International Conference on Automated Software Engineering : [proceedings] pp. 2024 - 2034
• Main Authors: Zhang, Man, Arcuri, Andrea, Teng, Piyun, Xue, Kaiming, Wang, Wenhao
• Format: Conference Proceeding
• Language: English
• Published: ACM 27.10.2024
• Subjects: Automated Test Case Generation; Fault detection; Fault diagnosis; Fuzzing; Glass box; Industries; Logic; Microservice architectures; Microservices; Prototypes; SBST; Software engineering; Testing
• ISSN: 2643-1572
• DOI: 10.1145/3691620.3695265

Microservices is now becoming a promising architecture to build large-scale web services in industry. Due to the high complexity of enterprise microservices, industry has an urgent need to have a solution to enable automated testing of such systems. EvoMaster is an open-source fuzzer, equipped with the state-of-the-art techniques for supporting automated system-level testing of Web APIs. It has been assessed as the most performant tool in two recent empirical studies in terms of line coverage and fault detection. In this paper, we carried out an empirical experiment to investigate how to better apply the state-of-the-art academic prototype (i.e., EvoMaster) in industrial context. We extended the tool to handle seeding of existing industrial tests, and mocking of external services with their data handled as part of the input fuzzing. We studied two configurations of EvoMaster, using two time budgets, on 40 enterprise RPC-based APIs (involving 5.6 million lines of code for their core business logic) at Meituan. Results show that, compared to existing practice of manual system-level testing and tests produced by record and replay of online traffic, EvoMaster demonstrates clear additional benefits. EvoMaster with the best configuration is capable of covering up to 32.4% line coverage, covering more than 10% line coverage on 36 out of 40 (90%) case studies, and identifying on average 3520 potential faults in these 40 APIs. In addition, we also identified and discussed important challenges in fuzzing enterprise microservices that must be addressed in the future.CCS CONCEPTS* Software and its engineering → Search-based software engineering; Software verification and validation.