# US 11,907,768 B2 · Entanglement of pages and guest threads > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,768 B2 - Document ID: US11907768B2 - Status: Granted - Kind: B2 - Application number: 17/028,874 - Filed: 2020-09-22 - Published: 2021-01-14 - Granted: 2024-02-20 - Earliest claimed: 2017-08-31 - Assignees: Hewlett Packard Enterprise Development LP - Applicants: Hewlett Packard Enterprise Development LP - Inventors: Isaac R. Nassi; David P. Reed; Mark Hill - Web page: https://patentgrep.com/patent/US11907768B2 ## CPC - G06F 9/5077: {Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455)} - G06F 9/3009: {Thread control instructions} - G06F 9/30123: {according to context, e.g. thread buffers} - G06F 9/3836: {Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing} - G06F 2009/45583: {Memory management, e.g. access or allocation} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Entanglement of pages and threads is disclosed. An indication is received of a stalling event caused by a requested portion of memory being inaccessible. It is determined that the requested portion of memory is an entangled portion of memory that is entangled with a physical node in a plurality of physical nodes. A type of the entangled portion of memory is determined. The stalling event is handled based at least in part on the determined type of the entangled portion of memory. ## Claims ### Claim 1 A system, comprising: a plurality of physically interconnected computing nodes comprising a first computing node and a second computing node separate from the first computing node; and a kernel that: receives an indication of a stalling event caused by the first computing node requesting access to a requested portion of memory and the requested portion of memory being inaccessible by the first computing node; determines that the requested portion of memory is an entangled portion of memory that is designated as being entangled with the second computing node; handles the stalling event, wherein the handling comprises: migrating the requested portion of memory to the second computing node, wherein the requested portion of memory is accessed by a virtual processor on the second computing node; and migrating the requested portion of memory back to the first computing node responsive to completion of the access by the virtual processor; determines a cost to migrate the requested portion of memory from the second computing node to the first computing node; and determines to migrate the requested portion of the memory to the second computing node based on the cost. ### Claim 2 (depends on 1) The system recited in claim 1, wherein the kernel determines that the entangled portion of memory is a portion of memory that is frequently accessed by a plurality of threads. ### Claim 3 (depends on 2) The system recited in claim 2, wherein the kernel, responsive to determining that the entangled portion of memory is a portion of memory that is frequently accessed by the plurality of threads, handles the stalling event. ### Claim 4 (depends on 1) The system of claim 1, wherein the kernel: evaluates timing measurements associated with page migration between the second computer node and the first computer node; and estimates the cost based on the timing measurements. ### Claim 5 (depends on 1) The system of claim 1, wherein the kernel: determines a frequency of write accesses by threads to the requested portion of memory; determines a number of the threads; and determines that the requested portion of memory is an entangled portion based on the number of writes and the number of threads. ### Claim 6 A method, comprising: receiving an indication of a stalling event caused by a first computing node of a plurality of physically interconnected computing nodes requesting access to a requested portion of memory, wherein the plurality of physically interconnected computing nodes further comprises a second computing node separate from the first computing node; determining that the requested portion of memory is an entangled portion of memory that is designated as being entangled with the second computing node; handling the stalling event, wherein the handling comprises: migrating the requested portion of memory to the second computing node; accessing, by a virtual processor on the second computing node, the requested portion of memory; and migrating the requested portion of memory back to the first computing node responsive to completion of the access by the virtual processor; determining a cost to migrate the requested portion of memory from the second computing node to the first computing node; and determining to migrate the requested portion of the memory to the second computing node based on the cost. ### Claim 7 (depends on 6) The method of claim 6, further comprising determining that the entangled portion of memory is a portion of memory frequently accessed by a plurality of threads. ### Claim 8 (depends on 7) The method of claim 7, further comprising handling the stalling event responsive to the determination that the entangled portion of memory is a portion of memory frequently accessed by a plurality of threads. ### Claim 9 (depends on 6) The method of claim 6, wherein determining the cost comprises evaluating timing measurements associated with page migration between the second computer node and the first computer node. ### Claim 10 A non-transitory computer readable storage medium that stores machine readable instructions that, when executed by a machine, cause the machine to: receive an indication of a stalling event caused by the first computing node requesting access to a requested portion of memory, and the request portion of memory being inaccessible by the first computing node; determine that the requested portion of memory is an entangled portion of memory that is designated as being entangled with a second computing node; handle the stalling event, wherein handling the stalling event comprises: moving a virtual processor from the first computing node to the second computing node to allow the virtual processor to execute a guest thread on the second computing node to perform the access to the requested portion of memory; and returning the virtual processor back to the first computing node responsive to completion of the access to the requested portion of memory; and determine that the requested portion of memory is an entangled portion responsive to a determination that the requested portion of memory is frequently accessed by a plurality of threads. ### Claim 11 (depends on 10) The storage medium of claim 10, wherein: the virtual processor, when on the first computing node, has a state; and the instructions, when executed by the machine, further cause the machine to move the state in its entirety to the second computing node such that the virtual processor, when on the second computing node, has the state. ### Claim 12 (depends on 10) The storage medium of claim 10, wherein: the virtual processor, when on the first computing node, has a state; and the instructions, when executed by the machine, further cause the machine to partially move the state in its entirety to the second computing node such that the virtual processor, when on the second computing node, has a subset of the state. ### Claim 13 (depends on 10) The storage medium of claim 10, wherein the instructions, when executed by the machine, further cause the machine to limit instruction execution by the virtual processor on the second computing node to an instruction to perform the access to the requested portion of the memory. ### Claim 14 (depends on 10) The storage medium of claim 10, wherein the instructions, when executed by the machine, further cause the machine to place the virtual processor on the second computing node in an instruction trap mode to limit instruction execution by the virtual processor on the second computing node. ### Claim 15 (depends on 10) The storage medium of claim 10, wherein the virtual processor on the first computing node has a set of registers, and the instructions, when executed by the machine, further cause the machine to further partially move the set of registers to the second computing node so that the virtual processor on the second computing node has a subset of the registers. ### Claim 16 (depends on 10) The storage medium of claim 10, wherein the instructions, when executed by the machine, further cause the machine to determine that the requested portion of memory is an entangled portion responsive to a determination that the requested portion of memory is subject to a replication restriction. ### Claim 17 (depends on 10) The storage medium of claim 10, wherein the instructions, when executed by the machine, further cause the machine to determine that the requested portion of memory is an entangled portion based on a number of writes to responsive to a number of write accesses to the requested portion of memory and a diversity of threads issuing the write accesses.