System and method for detecting malicious code in address space of a process

A system for detection of malware on a computer, the system comprising:

• a memory;
• and
• at least one processor coupled to the memory and configured to:
• detect a first process executed on the computer in association with an application;
• intercept at least one function call made by the first process to a second process, wherein the at least one function call made by the first process is configured to execute code from an address space associated with the second process;
• determine one or more attributes associated with the at least one function call;
• determine whether to perform malware analysis of the code associated with the at least one function call in the address space associated with the second process based on application of one or more heuristic rules to the one or more attributes, wherein the one or more heuristic rules indicate that the code is to be analyzed when one of:
• the first process is writing data containing a header into the second process, the first process started from a predetermined directory, or the second process is a trusted system process;
• upon determination to perform malware analysis of the code, determine whether the code in the address space is malicious and determine system functions that are executed by the code by comparing addresses of functions called by the code and addresses of functions loaded in the address space associated with the second process;
• and
• generate one or more application control rules that prevent calling the system functions by the first process in the address space of the second process.

A non-transitory, computer-readable medium storing computer-executable instructions for detection of malware on a computer, comprising instructions to:

• detect a first process executed on the computer in association with an application;
• intercept at least one function call made by the first process to a second process, wherein the at least one function call made by the first process is configured to execute code from an address space associated with the second process;
• determine one or more attributes associated with the at least one function call;
• determine whether to perform malware analysis of the code associated with the at least one function call in the address space associated with the second process based on application of one or more heuristic rules to the one or more attributes, wherein the one or more heuristic rules indicate that the code is to be analyzed when one of:
• the first process is writing data containing a header into the second process, the first process started from a predetermined directory, or the second process is a trusted system process;
• upon determination to perform malware analysis of the code, determine whether the code in the address space is malicious;
• and determining system functions that are executed by the code by comparing addresses of functions called by the code and addresses of functions loaded in the address space associated with the second process;
• and
• generating one or more application control rules that prevent calling the system functions by the first process in the address space of the second process.