Tail latency-based job offloading in load-balanced groups

A computer usable program product comprising a computer-readable storage medium, and program instructions stored on the computer-readable storage medium, the stored program instructions comprising:

• program instructions to determine a type of a request that is currently being processed at a data processing system;
• program instructions to select a distribution from a set of processing time distributions, the distribution forming a model that is applicable to the type;
• program instructions to compute a threshold point for the model, wherein a processing time that exceeds a processing time at the threshold point is regarded as exhibiting tail latency according to the model, tail latency comprising a delay in processing of the request due to a reason other than a utilization of a resource of the data processing system exceeding a threshold utilization and a size of a queue in the data processing system exceeding a threshold size;
• program instructions to evaluate that the request will experience tail latency during processing at the data processing system;
• program instruction to abort, responsive to the evaluating and prior to the request reaching the threshold, processing of the request at the data processing system;
• and
• program instructions to offload the request for processing at a peer data processing system in a load-balanced group of data processing systems;
• program instructions to identify a first queued request in the queue of the data processing system, the first queued request awaiting processing at the data processing system;
• program instructions to determine a first type of the first queued request;
• program instruction to select a first model from a first subset of the set of models, the first model corresponding to the first type;
• program instruction to estimate a first processing time of the first queued request using the first model;
• program instructions to evaluate that the first processing time will exceed a first acceptable processing time for the first queued request;
• program instructions to remove, responsive to the first processing time exceeding the first acceptable processing time, the first queued request from the queue;
• and
• program instructions to offload the first queued request for processing at a second peer data processing system in the load-balanced group of data processing systems.

A computer system comprising one or more processors, one or more computer-readable memories, and one or more computer-readable storage media, and program instructions stored on at least one of the one or more storage media for execution by at least one of the one or more processors via at least one of the one or more memories, the stored program instructions comprising:

• program instructions to determine a type of a request that is currently being processed at a data processing system;
• program instructions to select a distribution from a set of processing time distributions, the distribution forming a model that is applicable to the type;
• program instructions to compute a threshold point for the model, wherein a processing time that exceeds a processing time at the threshold point is regarded as exhibiting tail latency according to the model, tail latency comprising a delay in processing of the request due to a reason other than a utilization of a resource of the data processing system exceeding a threshold utilization and a size of a queue in the data processing system exceeding a threshold size;
• program instructions to evaluate that the request will experience tail latency during processing at the data processing system;
• program instructions to abort, responsive to the evaluating and prior to the request reaching the threshold, processing of the request at the data processing system;
• program instructions to offload the request for processing at a peer data processing system in a load-balanced group of data processing systems;
• program instructions to identify a first queued request in the queue of the data processing system, the first queued request awaiting processing at the data processing system;
• program instructions to determine a first type of the first queued request;
• program instruction to select a first model from a first subset of the set of models, the first model corresponding to the first type;
• program instruction to estimate a first processing time of the first queued request using the first model;
• program instructions to evaluate that the first processing time will exceed a first acceptable processing time for the first queued request;
• program instructions to remove, responsive to the first processing time exceeding the first acceptable processing time, the first queued request from the queue;
• and
• program instructions to offload the first queued request for processing at a second peer data processing system in the load-balanced group of data processing systems.