Automatically Identifying Trigger-based Behavior in Malware

• Published in: Botnet Detection pp. 65 - 88
• Main Authors: Brumley, David, Hartwig, Cody, Liang, Zhenkai, Newsome, James, Song, Dawn, Yin, Heng
• Format: Book Chapter
• Language: English
• Published: Boston, MA Springer US 2008
• Series: Advances in Information Security
• Subjects: Feasible Path; Intermediate Representation; Malicious Code; Symbolic Execution; Trigger Condition
• ISBN: 9780387687667; 0387687661
• ISSN: 1568-2633
• DOI: 10.1007/978-0-387-68768-1_4

Malware often contains hidden behavior which is only activated when properly triggered. Well known examples include: the MyDoom worm which DDoS’s on particular dates, keyloggers which only log keystrokes for particular sites, and DDoS zombies which are only activated when given the proper command. We call such behavior trigger-based behavior Currently, trigger-based behavior analysis is often performed in a tedious, manual fashion. Providing even a small amount of assistance would greatly assist and speed-up the analysis. In this chapter, we propose that automatic analysis of trigger-based behavior in malware is possible. In particular, we design an approach for automatic trigger-based behavior detection and analysis using dynamic binary instrumentation and mixed concrete and symbolic execution. Our approach shows that in many cases we can: (1) detect the existence of trigger-based behavior, (2) find the conditions that trigger such hidden behavior, and (3) find inputs that satisfy those conditions, allowing us to observe the triggered malicious behavior in a controlled environment. We have implemented MineSweeper, a system utilizing this approach. In our experiments, MineSweeper has successfully identified trigger-based behavior in real-world malware. Although there are many challenges presented by automatic trigger-based behavior detection, MineSweeper shows us that such automatic analysis is possible and encourages future work in this area