System and method for countering detection of emulation by malware

A system for investigating a presence of malicious code in an application program, the system comprising:

• computing hardware including at least one processor, memory, and an operating system that facilitates user-mode data processing and kernel-mode data processing wherein a plurality of system application program interfaces (APIs) provide access to user-mode and kernel-mode system functions, the computing hardware including instructions stored in a non-transitory medium that, when executed by the at least one processor, establish:
• a first emulator module that emulates instructions of the application program according to a first emulation mode wherein the instructions of the application program are carried out sequentially in a first virtual execution environment that represents the user-mode data processing of the operating system;
• an instruction review module that detects, among the instructions of the application program, a system API call requesting execution of a user-mode system function;
• a second emulator module that, in response to detecting the system API call, emulates instructions of the user-mode system function called by the API call according to a second emulation mode wherein the instructions of the user-mode system function are carried out sequentially in a second virtual execution environment that represents the user-mode data processing of the operating system, wherein certain processor and memory states affected by the instructions of the user-mode system function are represented;
• and
• an analyzer module that analyzes results of the emulating of the application program instructions according to the first emulation mode to detect any presence of malicious code in the application program.

A system for investigating a presence of malicious code in an application program, the system comprising:

• computing hardware including at least one processor, memory, and an operating system that facilitates user-mode data processing and kernel-mode data processing wherein a plurality of system application program interfaces (APIs) provide access to user-mode and kernel-mode system functions, the computing hardware including instructions stored in a non-transitory medium that, when executed by the at least one processor, establish:
• means for emulating instructions of the application program according to a first emulation mode wherein the instructions of the application program are carried out sequentially in a first virtual execution environment that represents the user-mode data processing of the operating system;
• means for detecting, among the instructions of the application program, a system API call requesting execution of a user-mode system function;
• means for emulating instructions of the user-mode system function called by the API call according to a second emulation mode wherein the instructions of the user-mode system function are carried out sequentially in a second virtual execution environment that represents the user-mode data processing of the operating system, wherein certain processor and memory states affected by the instructions of the user-mode system function are represented, said means for emulating instructions initiate operation in response to detecting the system API call;
• and
• means for analyzing results of the emulating of the application program instructions according to the first emulation mode to detect any presence of malicious code in the application program.