Controlling system load based on memory bandwidth

A system comprising:

• one or more processors;
• a plurality of computing devices;
• and
• one or more non-transitory computer readable storage media coupled to the one or more processors and storing instructions, which, when executed by the one or more processors, causes the one or more processors to perform operations comprising:
• sending input data to the plurality of computing devices configured to process the input data, wherein a respective portion of the input data is sent to each of the plurality of computing devices according to a load-balancing distribution;
• receiving, from a first computing device of the plurality of computing devices, data characterizing memory bandwidth for a memory device on the first computing device over a period of time, wherein the portion of the input data sent to the first computing device comprises a request to the first computing device to return output data by processing the data using a machine learning model;
• determining, at least from the data characterizing the memory bandwidth and a memory bandwidth saturation point for the first computing device, that the first computing device can process additional data within a predetermined latency threshold, the predetermined latency threshold specifying a maximum tolerated rate of change between access latency and the memory bandwidth, and the access latency being a measure of time to access data stored on the memory device, wherein the memory bandwidth saturation point for the first computing device is determined by:
• receiving measures of access latency at different memory bandwidths, each memory bandwidth corresponding to a respective measure of access latency, and
• identifying the memory bandwidth saturation point for the first computing device as a point where a graph of the access latency as a function of the memory bandwidth begins to exhibit a non-linear relationship such that an increase in access latency per unit increase of memory bandwidth is greater than another increase in access latency per unit increase of memory bandwidth exhibiting a linear relationship between the access latency and the memory bandwidth;
• identifying a first memory bandwidth corresponding to the memory bandwidth saturation point;
• and
• in response to the determining that the first computing device can process additional data within the predetermined latency threshold, sending the additional data to the first computing device for processing by the first computing device.

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• sending input data to a plurality of computing devices configured to process the input data, wherein a respective portion of the input data is sent to each of the plurality of computing devices according to a load-balancing distribution;
• receiving, from a first computing device of the plurality of computing devices, data characterizing memory bandwidth for a memory device on the first computing device and over a period of time, wherein the portion of the input data sent to the first computing device comprises a request to the first computing device to return output data by processing the data using a machine learning mode;
• determining, at least from the data characterizing the memory bandwidth and a memory bandwidth saturation point for the first computing device, that the first computing device can process additional data within a predetermined latency threshold, the predetermined latency threshold specifying a maximum tolerated rate of change between access latency and the memory bandwidth, and the access latency being a measure of time to access data stored on the memory device, wherein the memory bandwidth saturation point for the first computing device is determined by:
• receiving measures of access latency at different memory bandwidths, each memory bandwidth corresponding to a respective measure of access latency, and
• identifying the memory bandwidth saturation point for the first computing device as a point where a graph of the access latency as a function of the memory bandwidth begins to exhibit a non-linear relationship such that an increase in access latency per unit increase of memory bandwidth is greater than another increase in access latency per unit increase of memory bandwidth exhibiting a linear relationship between the access latency and the memory bandwidth;
• identifying a first memory bandwidth corresponding to the memory bandwidth saturation point;
• and
• in response to the determining that the first computing device can process additional data within the predetermined latency threshold, sending the additional data to the first computing device for processing by the first computing device.