Detecting return-oriented programming payloads by evaluating data for a gadget address space address and determining whether operations associated with instructions beginning at the address indicate a return-oriented programming payload

A method for detecting the presence of a return-oriented programming (ROP) payload in data, comprising:

• for each instruction of a plurality of instructions beginning at an address corresponding to a piece of data in the data, using a hardware processor to:
• attempt to execute the instruction;
• count the instruction as part of an instruction count;
• at least one of:
• determine that the instruction does not have an invalid execution address;
• determine that the instruction is not invalid;
• and determine that the instruction is not privileged;
• and
• determine that the instruction count is not greater than or equal to at least one threshold;
• in response to:
• the at least one of:
• determining that the instruction does not have an invalid execution address;
• determining that the instruction is not invalid;
• and determining that the instruction is not privileged;
• and
• the determining that the instruction count is not greater than or equal to the at least one threshold,
• using the hardware processor to increase a gadget count;
• and
• indicating, using the hardware processor, that an ROP payload is present in the data in response to the gadget count meeting a threshold greater than one.

A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for detecting the presence of a return-oriented programming (ROP) payload in data, the method comprising:

• for each instruction of a plurality of instructions beginning at an address corresponding to a piece of data in the data:
• attempting to execute the instruction;
• counting the instruction as part of an instruction count;
• at least one of:
• determining that the instruction does not have an invalid execution address;
• determining that the instruction is not invalid;
• and determining that the instruction is not privileged;
• and
• determining that the instruction count is not greater than or equal to at least one threshold;
• in response to:
• the at least one of:
• determining that the instruction does not have an invalid execution address;
• determining that the instruction is not invalid;
• and determining that the instruction is not privileged;
• and
• the determining that the instruction count is not greater than or equal to the at least one threshold,
• increasing a gadget count;
• and
• indicating that an ROP payload is present in the data in response to the gadget count meeting a threshold greater than one.