Control-flow integrity with managed code and unmanaged code

An apparatus with features to enforce control-flow integrity, the apparatus comprising:

• at least one machine-accessible medium;
• a binary translator in the machine-accessible medium;
• wherein the binary translator, when executed by a hardware processor in a data processing system, converts unmanaged code in the data processing system into hardened unmanaged code (HUC) by including control-flow integrity (CFI) features in the HUC;
• and
• wherein the CFI features comprise one or more CFI instructions which utilize hardware circuitry for CFI enforcement of the hardware processor for transfers of control initiated by the HUC;
• managed code in the at least one machine-accessible medium, wherein the managed code is configured (a) to execute within a managed runtime environment (MRE) in the data processing system and (b) to transfer control out from the MRE to the HUC, in response to a transfer control statement in the managed code;
• and
• a preprocessor in the machine-accessible medium, wherein the preprocessor, when executed, converts managed code in the data processing system into hardened managed code (HMC) by ensuring that the HMC includes CFI features,
• wherein, when transferring control from the MRE to the HUC, the HMC is configured to (i) use an MRE-to-unmanaged (MTU) CFI tag to transfer control out from the MRE to an exit-path code stub, wherein the exit-path code stub comprises a matching MTU CFI tag and (ii) to use an unmanaged CFI tag to transfer control from the exit-path code stub to a destination address in the HUC.

A method for enforcing control-flow integrity, the method comprising:

• using binary translation to convert unmanaged code in a data processing system into hardened unmanaged code (HUC) by including control-flow integrity (CFI) features in the HUC;
• using a preprocessor to convert managed code in the data processing system into hardened managed code (HMC) by ensuring that the HMC includes CFI features;
• executing the HMC in a managed runtime environment (MRE) in the data processing system;
• transferring control out from the MRE to the HUC, in response to a transfer control statement in the HMC;
• using the CFI features in the HUC for transfers of control initiated by the HUC, wherein the CFI features comprise one or more CFI instructions which utilize hardware circuitry for CFI enforcement of a hardware processor of the data processing system for the transfers of control initiated by the HUC;
• using dynamic binary translation to convert self-modifying code (SMC) produced by the HUC into hardened SMC (HSC) by including CFI features in the HSC;
• saving the HSC in a translation cache;
• when executing HUC that includes a transfer control statement which would cause a transfer to SMC, automatically determining whether the SMC has been changed since the SMC was translated into HSC;
• in response to a determination that the SMC has been changed, automatically updating the HSC in the translation cache to shadow the SMC;
• and
• after updating the HSC to shadow the SMC, transferring control from the HUC to the HSC.