System and method to improve nested virtual machine monitor performance

A system comprising:

• a memory;
• and
• a processor, communicatively coupled to the memory, comprising:
• a first register to store an invalidation mode flag associated with a virtual processor identifier (VPID);
• and
• a processing core, communicatively coupled to the first register, the processing core comprising a logic circuit to execute a virtual machine monitor (VMM) environment, the VMM environment comprising a root mode VMM supporting a non-root mode VMM, the non-root mode VMM to execute a virtual machine (VM) identified by the VPID, the logic circuit further comprising an invalidation circuit to:
• execute a virtual processor invalidation (INVVPID) instruction issued by the non-root mode VMM, the INVVPID instruction comprising a reference to an INVVPID descriptor that specifies a linear address and the VPID;
• and
• responsive to determining that the invalidation mode flag is set, invalidate, without triggering a VM exit event, a memory address mapping associated with the linear address.

A method comprising:

• executing, by a processing device, a virtual machine monitor (VMM) environment, the VMM environment comprising a root mode VMM supporting a non-root mode VMM, the non-root mode VMM executing a first virtual machine (VM) identified by a first virtual processor identifier (VPID), wherein the processing device comprises a first register to store an invalidation mode flag associated with the first VPID;
• executing a virtual processor invalidation (INVVPID) instruction issued by a non-root mode VMM, the INVVPID instruction comprising a reference to an INVVPID descriptor that specifies a linear address and the first VPID;
• and
• responsive to determining that the invalidation mode flag is set, invalidating, without triggering a VM exit event, a memory address mapping associated with the linear address.