System and method for branch function based obfuscation

A system comprising:

• a processor;
• a memory storing instructions for controlling the processor to perform steps comprising:
• receiving compiled code;
• determining a return address and destination value pair for each branch of a plurality of branches in the compiled code;
• based on the return address and the destination value for each branch, constructing a binary tree data structure with nodes and leaf nodes, each node storing a constant value to yield constant values, and each leaf node storing a destination value to yield destination values, wherein the constant values are arranged such that searching the binary tree data structure by return address leads to a corresponding destination value;
• inserting the binary tree data structure in the compiled code;
• and
• replacing each of the plurality of branches with instructions in the compiled code for performing a corresponding branching operation based on the binary tree data structure.

A non-transitory computer-readable storage medium storing instructions which, when executed by a computing device, cause the computing device to perform steps comprising:

• at a branch represented by a replacement branch instruction:
• determining a return address for the replacement branch instruction;
• searching a tree data structure, based on the return address, to determine a destination value corresponding to the return address, wherein the tree data structure does not contain the return address, and wherein a position of the destination value within the tree data structure is based on the return address;
• and
• executing the replacement branch instruction by jumping to the destination value, and, after executing additional instructions associated with the branch, jumping back to the return address.

A system comprising:

• a processor;
• a memory storing instructions for controlling the processor to perform, at a branch represented by a replacement branch instruction, steps comprising:
• determining a return address for the replacement branch instruction;
• searching a tree data structure, based on the return address, to determine a destination value corresponding to the return address, wherein the tree data structure does not contain the return address, and wherein a position of the destination value within the tree data structure is based on the return address;
• and
• executing the replacement branch instruction by jumping to the destination value, and, after executing additional instructions associated with the branch, jumping back to the return address.

A method comprising:

• receiving code including a plurality of branches;
• obfuscating, via a processor the plurality of branches via a hash table, based on a respective return address and destination value pair for at least some of the plurality of branches;
• identifying collisions in the hash table;
• for each collision in the hash table, inserting, via the processor, in a binary tree data structure a respective collision return address having a corresponding collision destination value, each internal node of the binary tree data structure storing a constant value to yield constant values, and each leaf node of the binary tree data structure storing a destination value to yield destination values, wherein the constant values are arranged such that searching the binary tree data structure by return address leads to a corresponding destination value;
• inserting, via the processor, the binary tree data structure in the code;
• and
• replacing, via the processor, branches corresponding to collisions with instructions in the code for performing a corresponding branching operation based on the binary tree data structure.