# US 12,117,931 B2 · Coordinated garbage collection in distributed systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,117,931 B2 - Document ID: US12117931B2 - Status: Granted - Kind: B2 - Application number: 18/469,480 - Filed: 2023-09-18 - Published: 2024-01-04 - Granted: 2024-10-15 - Earliest claimed: 2014-09-10 - Assignees: Oracle International Corporation - Applicants: Oracle International Corporation - Inventors: Timothy L. Harris; Martin C. Maas - Web page: https://patentgrep.com/patent/US12117931B2 ## CPC - G06F 12/0253: {Garbage collection, i.e. reclamation of unreferenced memory} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - G06F 9/522: {Barrier synchronisation} - G06F 11/301: {where the computing system is a virtual computing platform, e.g. logically partitioned systems (virtual machines G06F9/45533; logical partitioning of resources G06F9/5077)} - G06F 11/34: Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation {; Recording or statistical evaluation of user activity, e.g. usability assessment} - G06F 11/3409: {for performance assessment} - G06F 12/0276: {Generational garbage collection} - G06F 2009/45583: {Memory management, e.g. access or allocation} - G06F 2212/1024: Latency reduction - G06F 2212/152: Virtualized environment, e.g. logically partitioned system - G06F 2212/154: Networked environment - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Fast modern interconnects may be exploited to control when garbage collection is performed on the nodes (e.g., virtual machines, such as JVMs) of a distributed system in which the individual processes communicate with each other and in which the heap memory is not shared. A garbage collection coordination mechanism (a coordinator implemented by a dedicated process on a single node or distributed across the nodes) may obtain or receive state information from each of the nodes and apply one of multiple supported garbage collection coordination policies to reduce the impact of garbage collection pauses, dependent on that information. For example, if the information indicates that a node is about to collect, the coordinator may trigger a collection on all of the other nodes (e.g., synchronizing collection pauses for batch-mode applications where throughput is important) or may steer requests to other nodes (e.g., for interactive applications where request latencies are important). ## Claims ### Claim 1 A method, comprising: performing, by a plurality of hardware computing nodes interconnected via a network: configuring respective processes of a distributed application to execute on the plurality of hardware computing nodes, the configuring comprising setting a garbage collection policy of the distributed application to a garbage collection policy of a plurality of garbage collection policies including at least a parallel garbage collection policy and a staggered garbage collection policy; determining, during execution of the distributed application, that a computing node should perform a garbage collection operation, wherein the plurality of computing nodes comprise the computing node and one or more other computing nodes; and wherein responsive to the determining: performing, responsive to the garbage collection policy set to the parallel garbage collection policy, respective garbage collection operations at the computing node and the one or more other computing nodes; and performing, responsive to the garbage collection policy set to the staggered garbage collection policy, the garbage collection operation at the computing node while causing work directed to the computing node to be steered to at least one of the one or more other computing nodes not performing respective garbage collection operations. ### Claim 2 (depends on 1) The method of claim 1, wherein the distributed application is an application written in a garbage collected programming language, and wherein performing the garbage collection operation on a particular computing node comprises pausing or stopping execution of the distributed application on the particular computing node while the garbage collection operation is performed. ### Claim 3 (depends on 1) The method of claim 1, further comprising, responsive to the determining and the garbage collection policy set to the parallel garbage collection policy, sending a broadcast message indicating the start of the garbage collection operation to at least a portion of the plurality of computing nodes. ### Claim 4 (depends on 1) The method of claim 1, wherein the determining comprises determining a completion of a garbage collection interval. ### Claim 5 (depends on 1) The method of claim 1, wherein the determining comprises determining that a heap memory of one of the plurality of computing nodes meets a heap occupancy threshold. ### Claim 6 (depends on 1) The method of claim 1, further comprising, responsive to the determining and the garbage collection policy set to the staggered garbage collection policy, enforcing an upper bound on a number of computing nodes allowed to perform garbage collections at the same time. ### Claim 7 (depends on 6) The method of claim 6, wherein enforcing the upper bound on the number of computing nodes allowed to perform garbage collections at the same time comprises determining that a pool of zero or more tokens for granting garbage collections comprises at least one token, wherein performing the garbage collection operation at the computing node comprises allocating a token of the at least one token from the pool, and wherein the method further comprises returning the allocated token to the pool responsive to completion of performing the garbage collection operation at the computing node. ### Claim 8 One or more non-transitory computer-readable storage media storing program instructions that when executed on or across a plurality of hardware computing nodes cause the plurality of hardware computing nodes to perform: configuring respective processes of a distributed application to execute on the plurality of hardware computing nodes interconnected via a network, the configuring comprising setting a garbage collection policy of the distributed application to a garbage collection policy of a plurality of garbage collection policies including at least a parallel garbage collection policy and a staggered garbage collection policy; determining, during execution of the distributed application, that a computing node should perform a garbage collection operation, wherein the plurality of computing nodes comprise the computing node and one or more other computing nodes; and wherein responsive to the determining: performing, responsive to the garbage collection policy set to the parallel garbage collection policy, respective garbage collection operations at the computing node and the one or more other computing nodes; and performing, responsive to the garbage collection policy set to the staggered garbage collection policy, the garbage collection operation at the computing node while causing work directed to the computing node to be steered to at least one of the one or more other computing nodes not performing respective garbage collection operations. ### Claim 9 (depends on 8) The one or more non-transitory computer-readable storage media of claim 8, wherein the distributed application is an application written in a garbage collected programming language, and wherein performing the garbage collection operation on a particular computing node comprises pausing or stopping execution of the distributed application on the particular computing node while the garbage collection operation is performed. ### Claim 10 (depends on 8) The one or more non-transitory computer-readable storage media of claim 8, the plurality of hardware computing nodes further performing, responsive to the determining and the garbage collection policy set to the parallel garbage collection policy, sending a broadcast message indicating the start of the garbage collection operation to at least a portion of the plurality of computing nodes. ### Claim 11 (depends on 8) The one or more non-transitory computer-readable storage media of claim 8, wherein the determining comprises determining a completion of a garbage collection interval. ### Claim 12 (depends on 8) The one or more non-transitory computer-readable storage media of claim 8, wherein the determining comprises determining that a heap memory of one of the plurality of computing nodes meets a heap occupancy threshold. ### Claim 13 (depends on 8) The one or more non-transitory computer-readable storage media of claim 8, the plurality of hardware computing nodes further performing, responsive to the determining and the garbage collection policy set to the staggered garbage collection policy, enforcing an upper bound on a number of computing nodes allowed to perform garbage collections at the same time. ### Claim 14 (depends on 13) The one or more non-transitory computer-readable storage media of claim 13, wherein enforcing the upper bound on the number of computing nodes allowed to perform garbage collections at the same time comprises determining that a pool of zero or more tokens for granting garbage collections comprises at least one token, wherein performing the garbage collection operation at the computing node comprises allocating a token of the at least one token from the pool, and wherein the plurality of hardware computing nodes further perform returning the allocated token to the pool responsive to completion of performing the garbage collection operation at the computing node. ### Claim 15 A system, comprising: a plurality of hardware computing nodes interconnected via a network and individually comprising at least one processor and one or more heap memories, wherein individual nodes of the plurality of hardware computing nodes are configured to: configure respective processes of a distributed application to execute on the plurality of hardware computing nodes, the configuring comprising setting a garbage collection policy of the distributed application to a garbage collection policy of a plurality of garbage collection policies including at least a parallel garbage collection policy and a staggered garbage collection policy; determine, during execution of the distributed application, that a computing node should perform a garbage collection operation, wherein the plurality of computing nodes comprise the computing node and one or more other computing nodes; and wherein responsive to the determining: perform, responsive to the garbage collection policy set to the parallel garbage collection policy, respective garbage collection operations at the computing node and the one or more other computing nodes; and perform, responsive to the garbage collection policy set to the staggered garbage collection policy, the garbage collection operation at the computing node while causing work directed to the computing node to be steered to at least one of the one or more other computing nodes not performing respective garbage collection operations. ### Claim 16 (depends on 15) The system of claim 15, wherein the distributed application is an application written in a garbage collected programming language, and wherein performing the garbage collection operation on a particular computing node comprises pausing or stopping execution of the distributed application on the particular computing node while the garbage collection operation is performed. ### Claim 17 (depends on 15) The system of claim 15, wherein the individual nodes of the plurality of hardware computing nodes are further configured to, responsive to the determining and the garbage collection policy set to the parallel garbage collection policy, send a broadcast message indicating the start of the garbage collection operation to at least a portion of the plurality of computing nodes. ### Claim 18 (depends on 15) The system of claim 15, wherein the determining comprises determining a completion of a garbage collection interval. ### Claim 19 (depends on 15) The system of claim 15, wherein the determining comprises determining that a heap memory of one of the plurality of computing nodes meets a heap occupancy threshold. ### Claim 20 (depends on 15) The system of claim 15, wherein the individual nodes of the plurality of hardware computing nodes are further configured to, responsive to the determining and the garbage collection policy set to the staggered garbage collection policy, enforce an upper bound on a number of computing nodes allowed to perform garbage collections at the same time; wherein to enforce the upper bound on the number of computing nodes allowed to perform garbage collections at the same time, the individual nodes of the plurality of hardware computing nodes are further configured to determine that a pool of zero or more tokens for granting garbage collections comprises at least one token; wherein to perform the garbage collection operation at the computing node the individual nodes of the plurality of hardware computing nodes are further configured to allocate a token of the at least one token from the pool; and wherein the individual nodes of the plurality of hardware computing nodes are further configured to return the allocated token to the pool responsive to completion of performing the garbage collection operation at the computing node.