Workload balancing in a distributed storage system

A computer program product comprising:

• one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions executable by a computer to perform:
• determining, by a processor, a maximum bandwidth for a storage device included in a plurality of storage devices of a distributed storage system;
• determining, by the processor, a predicted bandwidth consumption for one or more frontend applications associated with the storage device during a time period based on a historical bandwidth consumption data that corresponds to the one or more frontend applications;
• determining, by the processor, a portion of the maximum bandwidth that is predicted to be available during the time period for a backend computational job for the storage device,
• determining, by the processor, whether a sum of (i) the portion of the maximum bandwidth and (ii) the predicted bandwidth consumption for one or more frontend applications is at most equal to the maximum bandwidth for the storage device;
• generating, by a processor, a data retrieval plan for the plurality of storage devices based on (i) whether the sum is at most equal to the maximum bandwidth for the storage device, (ii) the portion of the maximum bandwidth, (iii) the predicted bandwidth consumption for one or more frontend applications, (iv) a historical I/O access pattern for each given storage device of the plurality of storage devices, and (v) an expected available bandwidth for each given storage device of the plurality of storage devices during the time period;
• processing, by a processor, the backend computational job using the plurality of storage devices according to the data retrieval plan, wherein the data is retrieved from a given storage device of the plurality of storage devices, and wherein the given storage device is selected such that a sum of (i) a predicted bandwidth consumption for one or more frontend applications associated with the given storage device during the time period and (ii) a predicted bandwidth consumption for the backend computational job processing on the given storage device is (iii) at most equal to a maximum bandwidth for the given storage device;
• and
• throttling, by a processor, I/O throughput for the backend computational job according to the expected available bandwidth for the plurality of storage devices, wherein the I/O throughput is throttled to a throughput of greater than zero I/O instructions per second (IOPS), and wherein throttling comprises monitoring IOPS and suppressing the I/O throughput for the backend computational job when the IOPS exceeds a selected threshold.

A computer system comprising:

• one or more computer processors;
• one or more computer readable storage media;
• program instructions stored on the computer readable storage media for execution by at least one of the computer processors, the program instructions executable by a computer to perform:
• determining, by a processor, a maximum bandwidth for a storage device included in a plurality of storage devices of a distributed storage system;
• determining, by the processor, a predicted bandwidth consumption for one or more frontend applications associated with the storage device during a time period based on a historical bandwidth consumption data that corresponds to the one or more frontend applications;
• determining, by the processor, a portion of the maximum bandwidth that is predicted to be available during the time period for a backend computational job for the storage device,
• determining, by the processor, whether a sum of (i) the portion of the maximum bandwidth and (ii) the predicted bandwidth consumption for one or more frontend applications is at most equal to the maximum bandwidth for the storage device;
• generating, by a processor, a data retrieval plan for the plurality of storage devices based on (i) whether the sum is at most equal to the maximum bandwidth for the storage device, (ii) the portion of the maximum bandwidth, (iii) the predicted bandwidth consumption for one or more frontend applications, (iv) a historical I/O access pattern for each given storage device of the plurality of storage devices, and (v) an expected available bandwidth for each given storage device of the plurality of storage devices during the time period;
• processing, by a processor, the backend computational job using the plurality of storage devices according to the data retrieval plan, wherein the data is retrieved from a given storage device of the plurality of storage devices, and wherein the given storage device is selected such that a sum of (i) a predicted bandwidth consumption for one or more frontend applications associated with the given storage device during the time period and (ii) a predicted bandwidth consumption for the backend computational job processing on the given storage device is (iii) at most equal to a maximum bandwidth for the given storage device;
• and
• throttling, by a processor, I/O throughput for the backend computational job according to the expected available bandwidth for the plurality of storage devices, wherein the I/O throughput is throttled to a throughput of greater than zero I/O instructions per second (IOPS), and wherein throttling comprises monitoring IOPS and suppressing the I/O throughput for the backend computational job when the IOPS exceeds a selected threshold.