DeepReturn: A deep neural network can learn how to detect previously-unseen ROP payloads without using any heuristics

• Published in: Journal of computer security Vol. 28; no. 5; pp. 499 - 523
• Main Authors: Li, Xusheng, Hu, Zhisheng, Wang, Haizhou, Fu, Yiwei, Chen, Ping, Zhu, Minghui, Liu, Peng
• Format: Journal Article
• Language: English
• Published: Amsterdam IOS Press BV 01.01.2020
• Subjects: Artificial neural networks; Chains; Code reuse; Dismantling; Neural networks; Payloads; Run time (computers); Target detection
• ISSN: 0926-227X; 1875-8924
• DOI: 10.3233/JCS-191368

Return-oriented programming (ROP) is a code reuse attack that chains short snippets of existing code to perform arbitrary operations on target machines. Existing detection methods against ROP exhibit unsatisfactory detection accuracy and/or have high runtime overhead. In this paper, we present DeepReturn, which innovatively combines address space layout guided disassembly and deep neural networks to detect ROP payloads. The disassembler treats application input data as code pointers and aims to find any potential gadget chains, which are then classified by a deep neural network as benign or malicious. Our experiments show that DeepReturn has high detection rate (99.3%) and a very low false positive rate (0.01%). DeepReturn successfully detects all of the 100 real-world ROP exploits that are collected in-the-wild, created manually or created by ROP exploit generation tools. DeepReturn is non-intrusive and does not incur any runtime overhead to the protected program.