Selective memory deduplication for virtual machines

A system comprising:

• a memory;
• and
• a processing device, operatively coupled to the memory, execute a hypervisor to:
• receive a notification from a guest operating system (OS) of a virtual machine (VM), wherein the notification indicates an application access of a memory page;
• determine that the memory page is associated with a mergeable memory range for memory space of the guest OS, wherein the mergeable memory range is associated with guest OS memory pages to be deduplicated;
• map, in an extended page table (EPT) of the hypervisor, an EPT entry for the memory page to a memory address within the mergeable memory range, wherein the EPT entry indicates that the memory page is to be deduplicated;
• provide access to the memory page for the application;
• and
• perform a memory deduplication operation for the memory page in view of additional EPT entries for corresponding additional guest OS memory pages within the mergeable memory range.

A non-transitory computer readable medium comprising instructions, which when accessed by a processing device, cause the processing device to:

• receive, by a guest operating system (OS) of a virtual machine (VM), a first indication from an application to enable memory deduplication for a memory page associated with the application;
• identify a mergeable memory range for memory space of the guest OS, wherein the mergeable memory rage is associated with guest OS memory pages to be deduplicated;
• map, in a page table of the guest OS, a page table entry for the memory page to a memory address within the mergeable memory range;
• and
• cause a hypervisor to enable deduplication for the memory page responsive to detecting an access of the memory page by the application.