# US 12,117,936 B1 · Prefetch of microservices for incoming requests > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,117,936 B1 - Document ID: US12117936B1 - Status: Granted - Kind: B1 - Application number: 18/190,176 - Filed: 2023-03-27 - Published: 2024-10-03 - Granted: 2024-10-15 - Earliest claimed: 2023-03-27 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Christopher Neil Bailey; Matthew Paul Wilson; Ian Partridge; Sandra Hayward; David Richard Jones - Web page: https://patentgrep.com/patent/US12117936B1 ## CPC - G06F 12/0862: with prefetch - G06F 2212/1024: Latency reduction - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Prefetch of microservices for incoming requests. The method determines for an incoming request a Service Level Objective (SLO) requirement for latency of a request type of the incoming request. The method generates a set of possible microservice sequences for the request including a probability of occurrence of each of the possible microservice sequences and determines a set of prefetch permutations for the set of possible microservice sequences. A latency score is calculated for each prefetch permutation and any prefetch permutations that do not meet the SLO requirement for latency of the request type are eliminated. An optimal prefetch permutation of the remaining prefetch permutations is selected by considering a total cost of the prefetch permutation based on a cost of running each microservice in the set of sequences. ## Claims ### Claim 1 A computer-implemented method comprising: determining, for an incoming request, a Service Level Objective (SLO) requirement for latency of a request type of the incoming request; generating a set of possible microservice sequences for the request including a probability of occurrence of each of the possible microservice sequences; determining a set of prefetch permutations for the set of possible microservice sequences, wherein each prefetch permutation prefetches a different permutation of the microservices in the set of sequences; calculating a latency score for each prefetch permutation based on a total time of the sequence weighted by the probability of occurrence of the sequence, wherein the total time of the sequence is a sum of the startup latency of a microservice in the sequence when not prefetched and a compute time of the microservices in the sequence; eliminating any prefetch permutations that do not meet the SLO requirement for latency of the request type; and selecting a prefetch permutation of the remaining prefetch permutations by considering a total cost of the prefetch permutation based on a cost of running each microservice in the set of sequences. ### Claim 2 (depends on 1) The method of claim 1 wherein selecting the prefetch permutation balances the total cost with an overall latency of the prefetch permutation to minimize the combined total cost and overall latency. ### Claim 3 (depends on 1) The method of claim 1 including caching the permutation for a type of request and using the cached permutation for a subsequent incoming request of the type of request. ### Claim 4 (depends on 3) The method of claim 3 including updating the cached permutation based on updated costs and updated SLO when required. ### Claim 5 (depends on 3) The method of claim 3 including checking a cache for the permutation for a type of request of an incoming request. ### Claim 6 (depends on 1) The method of claim 1 wherein the probability of occurrence of each of the possible microservice sequences is based on historical request trace data. ### Claim 7 (depends on 1) The method as claimed in claim 1 wherein a cost of each microservice is determined based on resource usage and a platform type that the application is deployed on. ### Claim 8 (depends on 1) The method as claimed in claim 1 wherein the SLO requirement is a response being received within a defined latency period for a defined proportion of the total number of requests of the request type. ### Claim 9 (depends on 1) The method as claimed in claim 1 wherein eliminating any prefetch permutations that do not meet the SLO requirement for latency of the request type includes: determining the latency score for the permutation based on the total time of each microservice sequence weighted by the probability of occurrence of the sequence being used in the request, with the latency score being a percentage of the requests that meet the required latency of the SLO; and eliminating a permutation when the latency score does not meet the threshold percentage for the SLO requirements. ### Claim 10 (depends on 1) The method as claimed in claim 1 wherein selecting the prefetch permutation of the remaining prefetch permutations by considering a total cost of the prefetch permutation based on a cost of running each microservice in the set of sequences weighted by the probability of occurrence of the sequences. ### Claim 11 A computer system comprising: a processor(s) set; a machine readable storage device; and computer code stored on the machine readable storage device, with the computer code including instructions and data for causing the processor(s) set to perform operations including the following: determining, for an incoming request, a Service Level Objective (SLO) requirement for latency of a request type of the incoming request, generating a set of possible microservice sequences for the request including a probability of occurrence of each of the possible microservice sequences, determining a set of prefetch permutations for the set of possible microservice sequences, wherein each prefetch permutation prefetches a different permutation of the microservices in the set of sequences, calculating a latency score for each prefetch permutation based on a total time of the sequence weighted by the probability of occurrence of the sequence, wherein the total time of the sequence is a sum of the startup latency of a microservice in the sequence when not prefetched and a compute time of the microservices in the sequence, eliminating any prefetch permutations that do not meet the SLO requirement for latency of the request type, and selecting a prefetch permutation of the remaining prefetch permutations by considering a total cost of the prefetch permutation based on a cost of running each microservice in the set of sequences. ### Claim 12 (depends on 11) The computer system of claim 11 wherein the cost optimizing component balances the total cost with an overall latency of the prefetch permutation to minimize the combined total cost and overall latency. ### Claim 13 (depends on 11) The computer system of claim 11 further including: using a caching component for caching the permutation for a type of request; and using the cached permutation for a subsequent incoming request of the type of request. ### Claim 14 (depends on 13) The computer system of claim 13 further including: using a cache updating component for updating the cached permutation based on updated costs and updated SLO when required. ### Claim 15 (depends on 13) The computer system of claim 13 further including: using a cache checking component for checking a cache for the permutation for a type of request of an incoming request. ### Claim 16 (depends on 11) The computer system of claim 11 further including: using an historical request trace data component for accessing the probability of occurrence of each of the possible microservice sequences based on historical request trace data. ### Claim 17 (depends on 11) The computer system as claimed in claim 11 further including: using a cost accessing component for accessing a cost of each microservice based on resource usage and a platform type that the application is deployed on. ### Claim 18 (depends on 11) The computer system as claimed in claim 11 wherein the SLO requirement is a response being received within a defined latency period for a defined proportion of the total number of requests of the request type. ### Claim 19 (depends on 11) The computer system as claimed in claim 11 wherein a permutation eliminating component for eliminating any prefetch permutations that do not meet the SLO requirement for latency of the request type includes: a threshold component for eliminating a permutation when the latency score does not meet the threshold percentage for the SLO requirements. ### Claim 20 A computer program product comprising: a machine readable storage device; and computer code stored on the machine readable storage device, with the computer code including instructions and data for causing a processor(s) set to perform operations including the following: determining for an incoming request a Service Level Objective (SLO) requirement for latency of a request type of the incoming request, generating a set of possible microservice sequences for the request including a probability of occurrence of each of the possible microservice sequences, determining a set of prefetch permutations for the set of possible microservice sequences, wherein each prefetch permutation prefetches a different permutation of the microservices in the set of sequences, calculating a latency score for each prefetch permutation based on a total time of the sequence weighted by the probability of occurrence of the sequence, wherein the total time of the sequence is a sum of the startup latency of a microservice in the sequence when not prefetched and a compute time of the microservices in the sequence, eliminating any prefetch permutations that do not meet the SLO requirement for latency of the request type, and selecting a prefetch permutation of the remaining prefetch permutations by considering a total cost of the prefetch permutation based on a cost of running each microservice in the set of sequences.