Backward-Bounded DSE: Targeting Infeasibility Questions on Obfuscated Codes

• Published in: 2017 IEEE Symposium on Security and Privacy (SP) pp. 633 - 651
• Main Authors: Bardin, Sebastien, David, Robin, Marion, Jean-Yves
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.05.2017
• Subjects: Code analysis; Context; Deobfuscation; Malware; Reverse-Engineering; Robustness; Runtime; Symbolic Execution; Syntactics
• ISSN: 2375-1207
• DOI: 10.1109/SP.2017.36

Software deobfuscation is a crucial activity in security analysis and especially in malware analysis. While standard static and dynamic approaches suffer from well-known shortcomings, Dynamic Symbolic Execution (DSE) has recently been proposed as an interesting alternative, more robust than static analysis and more complete than dynamic analysis. Yet, DSE addresses only certain kinds of questions encountered by a reverser, namely feasibility questions. Many issues arising during reverse, e.g., detecting protection schemes such as opaque predicates, fall into the category of infeasibility questions. We present Backward-Bounded DSE, a generic, precise, efficient and robust method for solving infeasibility questions. We demonstrate the benefit of the method for opaque predicates and call stack tampering, and give some insight for its usage for some other protection schemes. Especially, the technique has successfully been used on state-of-the-art packers as well as on the government-grade X-Tunnel malware - allowing its entire deobfuscation. Backward-Bounded DSE does not supersede existing DSE approaches, but rather complements them by addressing infeasibility questions in a scalable and precise manner. Following this line, we propose sparse disassembly, a combination of Backward-Bounded DSE and static disassembly able to enlarge dynamic disassembly in a guaranteed way, hence getting the best of dynamic and static disassembly. This work paves the way for robust, efficient and precise disassembly tools for heavily-obfuscated binaries.