Probabilistic shellcode detection

A gateway comprising:

• an anti-malware engine operable to scan an arbitrary file and to determine if any candidate areas exist within the arbitrary file;
• for any given candidate area located within the arbitrary file, search at least one nearby area surrounding the candidate area for any instruction candidates;
• and
• for any such instruction candidates, calculate a statistical probability based on one or more disassembled instructions starting at a found offset of the instruction candidate to determine a likelihood that the arbitrary file includes shellcode.

A method of detecting shellcode in an arbitrary file comprising:

• scanning an arbitrary file to determine if any candidate areas exist within the arbitrary file;
• for any candidate areas found in the arbitrary file, first searching the areas surrounding the any candidate areas to determine if any function calls or any code branching instructions exist in the areas surrounding the any candidate areas;
• and
• if no function calls and no code branching instructions are found, searching the areas surrounding the any candidate areas for known characteristical shellcode sequences.

A computer network comprising:

• a gateway including an anti-malware engine operable to perform the following:
• receiving an arbitrary file;
• scanning the arbitrary file for repetitive constructs that are potentially intended to overflow a buffer;
• determining if any function calls or any code branching instructions exist in the areas surrounding the repetitive constructs;
• and
• generating a statistical probability representing the likelihood that the arbitrary file includes shellcode by performing a statistical analysis of the instructions starting at each found offset to generate an overall shellcode probability for the instructions starting at the each found offset.