Prefetch of microservices for incoming requests

A computer system comprising:

• a processor(s) set;
• a machine readable storage device;
• and
• computer code stored on the machine readable storage device, with the computer code including instructions and data for causing the processor(s) set to perform operations including the following:
• determining, for an incoming request, a Service Level Objective (SLO) requirement for latency of a request type of the incoming request,
• generating a set of possible microservice sequences for the request including a probability of occurrence of each of the possible microservice sequences,
• determining a set of prefetch permutations for the set of possible microservice sequences, wherein each prefetch permutation prefetches a different permutation of the microservices in the set of sequences,
• calculating a latency score for each prefetch permutation based on a total time of the sequence weighted by the probability of occurrence of the sequence, wherein the total time of the sequence is a sum of the startup latency of a microservice in the sequence when not prefetched and a compute time of the microservices in the sequence,
• eliminating any prefetch permutations that do not meet the SLO requirement for latency of the request type, and
• selecting a prefetch permutation of the remaining prefetch permutations by considering a total cost of the prefetch permutation based on a cost of running each microservice in the set of sequences.

A computer program product comprising:

• a machine readable storage device;
• and
• computer code stored on the machine readable storage device, with the computer code including instructions and data for causing a processor(s) set to perform operations including the following:
• determining for an incoming request a Service Level Objective (SLO) requirement for latency of a request type of the incoming request,
• generating a set of possible microservice sequences for the request including a probability of occurrence of each of the possible microservice sequences,
• determining a set of prefetch permutations for the set of possible microservice sequences, wherein each prefetch permutation prefetches a different permutation of the microservices in the set of sequences,
• calculating a latency score for each prefetch permutation based on a total time of the sequence weighted by the probability of occurrence of the sequence, wherein the total time of the sequence is a sum of the startup latency of a microservice in the sequence when not prefetched and a compute time of the microservices in the sequence,
• eliminating any prefetch permutations that do not meet the SLO requirement for latency of the request type, and
• selecting a prefetch permutation of the remaining prefetch permutations by considering a total cost of the prefetch permutation based on a cost of running each microservice in the set of sequences.