# US 12,112,212 B2 · Controlling system load based on memory bandwidth > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,212 B2 - Document ID: US12112212B2 - Status: Granted - Kind: B2 - Application number: 17/186,708 - Filed: 2021-02-26 - Published: 2022-09-01 - Granted: 2024-10-08 - Earliest claimed: 2021-02-26 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Dmytro Tymofieiev; Jaideep Singh; Kusum Kumar Madarasu - Web page: https://patentgrep.com/patent/US12112212B2 ## CPC - G06F 9/5083: {Techniques for rebalancing the load in a distributed system} - G06F 9/5016: {the resource being the memory} - G06F 11/3433: {for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029)} - G06F 11/3442: {for planning or managing the needed capacity} - G06F 12/0238: {Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory} - H04L 41/0896: Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities (flow or congestion control using dynamic resource allocation, e.g. in-call renegotiation, H04L47/76) - G06F 9/505: {considering the load} - G06F 11/3409: {for performance assessment} - G06F 2209/508: Monitor - G06F 2212/1024: Latency reduction - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and apparatus, including computer-readable storage media for load. A load balancer can input data to the plurality of computing devices configured to process the input data according to a load-balancing distribution. The load balancer can receive from a first computing device of the plurality of computing devices, data characterizing memory bandwidth for a memory device on the first computing device and over a period of time. The load balancer can determine, at least from the data characterizing the memory bandwidth and a memory bandwidth saturation point for the first computing device, that the first computing device can process additional data within a predetermined latency threshold. In response to the determining, the load balancer can send the additional data to the first computing device. ## Claims ### Claim 1 A method, comprising: sending, by one or more processors, input data to a plurality of computing devices configured to process the input data, wherein a respective portion of the input data is sent to each of the plurality of computing devices according to a load-balancing distribution; receiving, from a first computing device of the plurality of computing devices, data characterizing memory bandwidth for a memory device on the first computing device over a period of time, wherein the portion of the input data sent to the first computing device comprises a request to the first computing device to return output data by processing the data using a machine learning model; determining, at least from the data characterizing the memory bandwidth and a memory bandwidth saturation point for the first computing device, that the first computing device can process additional data within a predetermined latency threshold, the predetermined latency threshold specifying a maximum tolerated rate of change between access latency and the memory bandwidth, and the access latency being a measure of time to access data stored on the memory device, wherein the memory bandwidth saturation point for the first computing device is determined by: receiving measures of access latency at different memory bandwidths, each memory bandwidth corresponding to a respective measure of access latency, and identifying the memory bandwidth saturation point for the first computing device as a point where a graph of the access latency as a function of the memory bandwidth begins to exhibit a non-linear relationship such that an increase in access latency per unit increase of memory bandwidth is greater than another increase in access latency per unit increase of memory bandwidth exhibiting a linear relationship between the access latency and the memory bandwidth; identifying a first memory bandwidth corresponding to the memory bandwidth saturation point; and in response to determining that the first computing device can process additional data within the predetermined latency threshold, sending the additional data to the first computing device for processing by the first computing device. ### Claim 2 (depends on 1) The method of claim 1, wherein the memory bandwidth for the first computing device comprises measurements of a rate of data transferred between a processor and the memory device during each of a plurality of time intervals within the time period. ### Claim 3 (depends on 1) The method of claim 1, wherein the memory bandwidths higher than the first memory bandwidth correspond to measures of access latency that exceed the predetermined latency threshold. ### Claim 4 (depends on 1) The method of claim 1, wherein the first memory bandwidth is less than a maximum memory bandwidth attainable for the computing device. ### Claim 5 (depends on 1) The method of claim 1, wherein the predetermined latency threshold is at least partially based on a service level objective defining a maximum delay for the first computing device to respond to the request. ### Claim 6 (depends on 1) The method of claim 1, wherein the first computing device comprises one or more first computing device processors and/or one or more other memory devices, and wherein determining that the first computing device can receive additional data further comprises determining that the first computing device can receive additional data based on respective performance metrics, other than the memory bandwidth saturation point, for one or more of the first computing device processors and/or the other memory devices. ### Claim 7 (depends on 1) The method of claim 1, wherein the method further comprises: determining that the first computing device cannot process additional data within the predetermined latency threshold, and in response, sending the additional data to a second computing device different from the first computing device. ### Claim 8 A system comprising: one or more processors; a plurality of computing devices; and one or more non-transitory computer readable storage media coupled to the one or more processors and storing instructions, which, when executed by the one or more processors, causes the one or more processors to perform operations comprising: sending input data to the plurality of computing devices configured to process the input data, wherein a respective portion of the input data is sent to each of the plurality of computing devices according to a load-balancing distribution; receiving, from a first computing device of the plurality of computing devices, data characterizing memory bandwidth for a memory device on the first computing device over a period of time, wherein the portion of the input data sent to the first computing device comprises a request to the first computing device to return output data by processing the data using a machine learning model; determining, at least from the data characterizing the memory bandwidth and a memory bandwidth saturation point for the first computing device, that the first computing device can process additional data within a predetermined latency threshold, the predetermined latency threshold specifying a maximum tolerated rate of change between access latency and the memory bandwidth, and the access latency being a measure of time to access data stored on the memory device, wherein the memory bandwidth saturation point for the first computing device is determined by: receiving measures of access latency at different memory bandwidths, each memory bandwidth corresponding to a respective measure of access latency, and identifying the memory bandwidth saturation point for the first computing device as a point where a graph of the access latency as a function of the memory bandwidth begins to exhibit a non-linear relationship such that an increase in access latency per unit increase of memory bandwidth is greater than another increase in access latency per unit increase of memory bandwidth exhibiting a linear relationship between the access latency and the memory bandwidth; identifying a first memory bandwidth corresponding to the memory bandwidth saturation point; and in response to the determining that the first computing device can process additional data within the predetermined latency threshold, sending the additional data to the first computing device for processing by the first computing device. ### Claim 9 (depends on 8) The system of claim 8, wherein the first computing device comprises one or more general-purpose processors, one or more memory devices, and one or more hardware accelerators; and wherein the first computing device is configured to: receive the input data and, process the input data through the machine learning model, wherein processing the input data comprises executing a first portion of a machine learning model using the one or more general-purpose processors, and executing a second portion of the machine learning model different from the first portion and using the one or more hardware accelerators. ### Claim 10 (depends on 8) The system of claim 8, wherein the memory bandwidth for the first computing device comprises measurements of a rate of data transferred between a processor and the memory device during each of a plurality of time intervals within the time period. ### Claim 11 (depends on 8) The system of claim 8, wherein the memory bandwidths higher than the first memory bandwidth correspond to measures of access latency that exceed the predetermined latency threshold. ### Claim 12 (depends on 8) The system of claim 8, wherein the first memory bandwidth is less than a maximum memory bandwidth attainable for the computing device. ### Claim 13 (depends on 8) The system of claim 8, wherein the predetermined latency threshold is at least partially based on a service level objective defining a maximum delay for the first computing device to respond to the request. ### Claim 14 (depends on 8) The system of claim 8, wherein the first computing device comprises one or more first computing device processors and/or one or more other memory devices, and wherein determining that the first computing device can receive additional data further comprises determining that the first computing device can receive additional data based on respective performance metrics, other than the memory bandwidth saturation point, for one or more of the first computing device processors and/or the other memory devices. ### Claim 15 (depends on 8) The system of claim 8, wherein the method further comprises: determining that the first computing device cannot process additional data within the predetermined latency threshold, and in response, sending the additional data to a second computing device different from the first computing device. ### Claim 16 One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: sending input data to a plurality of computing devices configured to process the input data, wherein a respective portion of the input data is sent to each of the plurality of computing devices according to a load-balancing distribution; receiving, from a first computing device of the plurality of computing devices, data characterizing memory bandwidth for a memory device on the first computing device and over a period of time, wherein the portion of the input data sent to the first computing device comprises a request to the first computing device to return output data by processing the data using a machine learning mode; determining, at least from the data characterizing the memory bandwidth and a memory bandwidth saturation point for the first computing device, that the first computing device can process additional data within a predetermined latency threshold, the predetermined latency threshold specifying a maximum tolerated rate of change between access latency and the memory bandwidth, and the access latency being a measure of time to access data stored on the memory device, wherein the memory bandwidth saturation point for the first computing device is determined by: receiving measures of access latency at different memory bandwidths, each memory bandwidth corresponding to a respective measure of access latency, and identifying the memory bandwidth saturation point for the first computing device as a point where a graph of the access latency as a function of the memory bandwidth begins to exhibit a non-linear relationship such that an increase in access latency per unit increase of memory bandwidth is greater than another increase in access latency per unit increase of memory bandwidth exhibiting a linear relationship between the access latency and the memory bandwidth; identifying a first memory bandwidth corresponding to the memory bandwidth saturation point; and in response to the determining that the first computing device can process additional data within the predetermined latency threshold, sending the additional data to the first computing device for processing by the first computing device. ### Claim 17 (depends on 16) The computer-readable storage media of claim 16, wherein the memory bandwidth for the first computing device comprises measurements of a rate of data transferred between a processor and the memory device during each of a plurality of time intervals within the time period. ### Claim 18 (depends on 16) The computer-readable storage media of claim 16, wherein the memory bandwidths higher than the first memory bandwidth correspond to measures of access latency that exceed the predetermined latency threshold.