Optimizing tail latency via workload and resource redundancy in cloud

A computer program product for performing computing processing, the computer program product comprising a non-transitory computer readable storage having program instructions embodied therewith, the program instructions executable by a computer, to cause the computer to perform a method comprising:

• receiving a plurality of requests for computing processing;
• replicating at least some of the plurality of requests, wherein the requests are replicated based on a fractional replication factor;
• transmitting each received request and each replicated request to a computer resource for processing;
• determining a tail latency for performing the requested computing processing;
• and
• incrementally increasing the fractional replication factor when the determined tail latency does not meet a target tail latency;
• wherein the replicating is performed by a process selected from a group of processes consisting of:
• replicating selected requests, wherein the requests are selected based on a replication factor;
• replicating requests a number of times on average based on a randomly generated quantity;
• replicating each request with a different frequency;
• replicating requests stochastically by building a probability distribution having only integer values with an average equal to the fractional replication factor and determining a replication factor for each request based on the probability distribution;
• and
• replicating requests deterministically by defining a sequence of replication factors with an average equal to the fractional replication factor and determining a replication factor for each request based on the sequence.

A system for performing computing processing, the system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform:

• receiving a plurality of requests for computing processing;
• replicating at least some of the plurality of requests, wherein the requests are replicated based on a fractional replication factor;
• transmitting each received request and each replicated request to a computer resource for processing, determining a tail latency for performing the requested computing processing;
• and
• incrementally increasing the fractional replication factor when the determined tail latency does not meet a target tail latency;
• wherein the replicating is performed by a process selected from a group of processes consisting of:
• replicating selected requests, wherein the requests are selected based on a replication factor;
• replicating requests a number of times on average based on a randomly generated quantity;
• replicating each request with a different frequency;
• replicating requests stochastically by building a probability distribution having only integer values with an average equal to the fractional replication factor and determining a replication factor for each request based on the probability distribution;
• and
• replicating requests deterministically by defining a sequence of replication factors with an average equal to the fractional replication factor and determining a replication factor for each request based on the sequence.

A system for performing computing processing, comprising:

• a plurality of computing resources adapted to perform computing processing;
• and
• a load replicator adapted to:
• receive a plurality of requests for computing processing, replicate at least some of the plurality of requests, wherein the requests are replicated based on a fractional replication factor, transmit each received request and each replicated request to a computer resource for processing, determine a tail latency for performing the requested computing processing;
• and incrementally increase the fractional replication factor when the determined tail latency does not meet a target tail latency;
• wherein the replicating is performed by a process selected from a group of processes consisting of:
• replicating selected requests, wherein the requests are selected based on a replication factor;
• replicating requests a number of times on average based on a randomly generated quantity;
• replicating each request with a different frequency;
• replicating requests stochastically by building a probability distribution having only integer values with an average equal to the fractional replication factor and determining a replication factor for each request based on the probability distribution;
• and
• replicating requests deterministically by defining a sequence of replication factors with an average equal to the fractional replication factor and determining a replication factor for each request based on the sequence.