System and Method for Detecting Malware Using Isolated Environment

A system for detecting malicious applications, the system comprising:

• a memory operable to contain an isolated environment for storing a library of handler functions, wherein the handler functions are configured to control access of one or more applications to protected resources on a computer device;
• a processor coupled to the memory and configured to:
• modify the one or more applications to access the library of handler functions instead of corresponding application program interface (API) functions of the computer device and store the one or more applications in the isolated environment;
• receive by the handler functions API function calls from a modified application;
• analyze the received API function calls for malicious behavior characteristics;
• when the API function calls do not exhibit malicious behavior characteristics, the processor is configured to perform via the handler functions the API function calls to the protected resources;
• and
• when the API function calls exhibit malicious behavior characteristics, the processor is configured to perform one or more actions preventing access of the modified application to the protected resources.

A computer program product stored on a non-transitory computer-readable storage medium, the computer program product comprising computer-executable instructions for detecting malicious applications, including instructions for:

• providing a library of handler functions stored in an isolated environment, wherein the handler functions control access of one or more applications to protected resources on a computer device;
• modifying the one or more applications to access the library of handler functions instead of corresponding application program interface (API) functions of the computer device and storing the one or more applications in the isolated environment;
• receiving by the handler functions API function calls from a modified application;
• analyzing the received API function calls for malicious behavior characteristics;
• when the API function calls do not exhibit malicious behavior characteristics, performing by the handler functions the API function calls to the protected resources;
• and
• when the API function calls exhibit malicious behavior characteristics, performing one or more actions preventing access of the modified application to the protected resources.