Detecting a return-oriented programming exploit

#The present invention relates to the field of detecting a Return-Oriented Programming Exploit.

#Data Execution Prevention (DEP) is a feature used by some operating systems, including Microsoft Windows® that is used to reduce the risks of exploits that rely on, for example, storing code to stack or heap memory locations that normally do not contain executable code. DEP typically works by marking some regions of a computer readable memory as “non-executable”. This ensures that those regions can only store data that are recognized as being non-executable. A successful exploitation is therefore rendered less likely to happen as it may try to run code from a region of the memory that is marked as non-executable. The operating system detects this and prevents the code from being executed.

#In order to attempt to bypass DEP, some forms of exploit use Return-Oriented Programming (ROP). In one example, the exploit relies on overwriting a return address in the memory stack that has called by a valid application so that it points to code other than code associated with the normal application execution path. This code can be used for malicious purposes. An additional portion of the stack is overwritten that allows the exploit to call pre-existing functions without needing to inject malicious code into the program. This type of attack only uses existing executable code. ROP exploits can be used to chain individual small attacks. In other words, the stack memory is used indirectly to execute previously selected instructions (referred to as gadgets). Each gadget typically ends with the x86 subroutine return instruction (RET), which transfers the execution to the next gadget or to the payload. As all of the executed instructions are executed from executable memory areas within the original application, DEP is ineffective. The gadgets typically include code that is equivalent to calling a VirtualProtect API to enable execution rights for the memory page where the payload resides, which effectively renders DEP ineffective.

#FIG. 1 illustrates schematically in a data stack;

#FIG. 2 illustrates schematically in a block diagram a computer device according to an embodiment of the invention;

#FIG. 3 is a flow diagram showing steps according to an embodiment of the invention; and

#In order to better understand the invention, it is helpful to consider a known stack data structure as illustrated in FIG. 1. A stack is an abstract data structure that operates using a last in, first out principle. Two operations can be performed on a stack; a push operation that adds data to the stack, hiding any items already in the stack, and a pop operation that removes an item from the stack and returns it to a caller. A pop operation reveals previously concealed data. In the example of FIG. 1, four frames are shown (denoted by N, N-1, N-2 and N-3), and the remaining space is available stack space. The active frame N includes data and a location address for the previous frame, N-1, and is located by a stack pointer (ESP). A push operation adds another set of data to the stack in the available space at the location pointed to by the ESP, and the value of the ESP is decreased to the new stack pointer location. Furthermore, frame N becomes frame N-1. A pop operation removes frame N from the stack, revealing frame N-1 (which now becomes frame N) and changing the stack pointer value to 8. A ret operation gets a value from the ESP address into the EIP and starts executing that EIP address. It then pops a return address from the stack.

#Normally when an application calls a vulnerable subroutine, the call instruction that performs the control transfer to the subroutine stores a return address into the stack. The return address points to a location after the just-called instruction call to the subroutine. At the end of the subroutine there is normally an execution of ret instruction that transfers the execution back to the caller, which typically pops the stored return address from the stack to the EIP register.

#If an exploit is successfully able to modify the return address of the subroutine to point elsewhere than to the caller, for example it now points to a function named VirtualProtect. If we have hooked VirtualProtect we can compare the address of VirtualProtect to the address in the freed stack space (it is called freed stack space as the ret instruction changes the value(s) of the ESP register to indicate it is freed). If the compared addresses are the same then ROP is assumed to be in use.