Enhancing Dynamic Symbolic Execution by Automatically Learning Search Heuristics

• Published in: IEEE transactions on software engineering Vol. 48; no. 9; pp. 3640 - 3663
• Main Authors: Cha, Sooyoung, Hong, Seongjoon, Bak, Jiseong, Kim, Jingyoung, Lee, Junhee, Oh, Hakjoo
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.09.2022; IEEE Computer Society
• Subjects: Algorithms; concolic testing; Dynamic symbolic execution; execution-generated testing; Heuristic; Heuristic algorithms; Learning; Open source software; search heuristics; Search problems; Searching; Software algorithms; Software testing; Testing
• ISSN: 0098-5589; 1939-3520
• DOI: 10.1109/TSE.2021.3101870

We present a technique to automatically generate search heuristics for dynamic symbolic execution. A key challenge in dynamic symbolic execution is how to effectively explore the program's execution paths to achieve high code coverage in a limited time budget. Dynamic symbolic execution employs a search heuristic to address this challenge, which favors exploring particular types of paths that are most likely to maximize the final coverage. However, manually designing a good search heuristic is nontrivial and typically ends up with suboptimal and unstable outcomes. The goal of this paper is to overcome this shortcoming of dynamic symbolic execution by automatically learning search heuristics. We define a class of search heuristics, namely a parametric search heuristic, and present an algorithm that efficiently finds an optimal heuristic for each subject program. Experimental results with industrial-strength symbolic execution tools (e.g., KLEE) show that our technique can successfully generate search heuristics that significantly outperform existing manually-crafted heuristics in terms of branch coverage and bug-finding.