Tracing Stored Program Counter to Detect Polymorphic Shellcode

• Published in: IEICE Transactions on Information and Systems Vol. E91.D; no. 8; pp. 2192 - 2195
• Main Authors: KIM, Daewon, KIM, Ikkyun, OH, Jintae, JANG, Jongsoo
• Format: Journal Article
• Language: English
• Published: Oxford The Institute of Electronics, Information and Communication Engineers 2008; Oxford University Press
• Subjects: Applied sciences; Coding, codes; Cryptography; Exact sciences and technology; Information, signal and communications theory; network; Networks; polymorphism; Registers; Routines; security; Security systems; shellcode; Signal and communications theory; Signatures; Stacks; Stores; Telecommunications and information theory; Networked system; security; shellcode; Coding; Security of data; polymorphism; Encryption; Decryption; network
• ISSN: 0916-8532; 1745-1361
• DOI: 10.1093/ietisy/e91-d.8.2192

The shellcode use of the polymorphic form has become active as the de facto method for avoiding signature based network security system. We present a new static analysis method for detecting the decryption routine of the polymorphic shellcode. This method traces the processes by which the decryption routine stores the current program counter in a stack, moves the value between registers and uses the value in order to make the address of the encrypted code accessible. Most of decryption routines have the feature which they use the program counter stored on a stack as the address for accessing the memory that the encrypted code is positioned.