# US 2024/0202552 A1 · Using a logical tree structure to identify a foundation model inferencing server for fulfilling an inferencing request

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202552 A1
- Document ID: US20240202552A1
- Status: Published application
- Kind: A1
- Application number: 18/083,300
- Filed: 2022-12-16
- Published: 2024-06-20
- Earliest claimed: 2022-12-16
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Mudhakar Srivatsa; Satishkumar Sadagopan; Utpal Mangla; Dinesh C. Verma; Gerald Coon; Mathews Thomas
- Web page: https://patentgrep.com/patent/US20240202552A1

## CPC

- G06N 5/04: Inference or reasoning models
- G06F 9/5033: {considering data affinity}
- G06F 9/5044: {considering hardware capabilities}
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/00: Machine learning
- G06F 2209/501: Performance criteria
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer-implemented method, according to one embodiment, includes determining a plurality of downstream task models of a foundation model, and arranging the downstream task models into a logical tree structure. Each node of the logical tree structure represents a sequence of layers of an associated one of the downstream task models. In response to a determination that a request for inferencing on a target model has resulted in a cache miss occurring, the logical tree structure is used to identify an inferencing server that satisfies at least a first predetermined prerequisite for fulfilling the inferencing request. The method further includes causing the identified inferencing server to fulfill the inferencing request. A computer program product, according to one embodiment, includes a computer readable storage medium having program instructions embodied therewith. The program instructions are readable and/or executable by a computer to cause the computer to perform the foregoing method.

## Claims

### Claim 1

A computer-implemented method, comprising:
determining a plurality of downstream task models of a foundation model;
arranging the downstream task models into a logical tree structure, wherein each node of the logical tree structure represents a sequence of layers of an associated one of the downstream task models;
in response to a determination that a request for inferencing on a target model has resulted in a cache miss occurring, using the logical tree structure to identify an inferencing server that satisfies at least a first predetermined prerequisite for fulfilling the inferencing request; and
causing the identified inferencing server to fulfill the inferencing request.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the cache miss occurs based on the target model not being fully loaded on any inferencing servers that are considered as candidates for the inferencing request.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the first predetermined prerequisite specifies that a residual capacity of an inferencing server be capable of loading a remainder of the target model.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, comprising: in response to the determination that the inferencing request has resulted in the cache miss occurring, using the logical tree structure to identify an inferencing server that satisfies the first predetermined prerequisite for fulfilling the inferencing request and a second predetermined prerequisite for fulfilling the inferencing request, wherein the second predetermined prerequisite specifies that an inferencing server have a relatively larger portion of the target model pre-loaded than any of the other inferencing servers that are considered as candidates for the inferencing request.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein a plurality of the inferencing servers that are considered as candidates for the inferencing request are determined to satisfy the first predetermined prerequisite and the second predetermined prerequisite, and comprising: applying different weightage values to a determination of which of the plurality of interfacing servers to use to fulfill the inferencing request.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the first predetermined prerequisite specifies that an inferencing server have a relatively larger portion of the target model pre-loaded than any of the other inferencing servers that are considered as candidates for the inferencing request, wherein the identified inferencing server is determined to have a relatively largest portion of the target model pre-loaded.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein arranging the downstream task models into the logical tree structure includes: using a common node within the logical tree structure to represent a common sequence of layers of the downstream task models, and using unique nodes within the logical tree structure to represent unique sequences of layers of the downstream task models.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the identified inferencing server is caused to load only a portion of the target model to fulfilling the inferencing request.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, comprising: obtaining an inference generated from the identified inferencing server; and storing the inference to a predetermined table of a hardware memory module.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the downstream task models are selected from the group consisting of: natural language processing (NLP) task models, sentiment analysis task models, and questioning and answering task models.

### Claim 11

A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable and/or executable by a computer to cause the computer to:
determine, by the computer, a plurality of downstream task models of a foundation model;
arrange, by the computer, the downstream task models into a logical tree structure, wherein each node of the logical tree structure represents a sequence of layers of an associated one of the downstream task models;
in response to a determination that a request for inferencing on a target model has resulted in a cache miss occurring, use, by the computer, the logical tree structure to identify an inferencing server that satisfies at least a first predetermined prerequisite for fulfilling the inferencing request; and
cause, by the computer, the identified inferencing server to fulfill the inferencing request.

### Claim 12 (depends on 11)

The computer program product of claim 11, wherein the cache miss occurs based on the target model not being fully loaded on any inferencing servers that are considered as candidates for the inferencing request.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein the first predetermined prerequisite specifies that a residual capacity of an inferencing server be capable of loading a remainder of the target model.

### Claim 14 (depends on 13)

The computer program product of claim 13, the program instructions readable and/or executable by the computer to cause the computer to: in response to the determination that the inferencing request has resulted in the cache miss occurring, use, by the computer, the logical tree structure to identify an inferencing server that satisfies the first predetermined prerequisite for fulfilling the inferencing request and a second predetermined prerequisite for fulfilling the inferencing request, wherein the second predetermined prerequisite specifies that an inferencing server have a relatively larger portion of the target model pre-loaded than any of the other inferencing servers that are considered as candidates for the inferencing request.

### Claim 15 (depends on 14)

The computer program product of claim 14, wherein a plurality of the inferencing servers that are considered as candidates for the inferencing request are determined to satisfy the first predetermined prerequisite and the second predetermined prerequisite, and the program instructions readable and/or executable by the computer to cause the computer to: apply, by the computer, different weightage values to a determination of which of the plurality of interfacing servers to use to fulfill the inferencing request.

### Claim 16 (depends on 11)

The computer program product of claim 11, wherein the first predetermined prerequisite specifies that an inferencing server have a relatively larger portion of the target model pre-loaded than any of the other inferencing servers that are considered as candidates for the inferencing request, wherein the identified inferencing server is determined to have a relatively largest portion of the target model pre-loaded.

### Claim 17 (depends on 11)

The computer program product of claim 11, wherein arranging the downstream task models into the logical tree structure includes: using a common node within the logical tree structure to represent a common sequence of layers of the downstream task models, and using unique nodes within the logical tree structure to represent unique sequences of layers of the downstream task models.

### Claim 18 (depends on 11)

The computer program product of claim 11, wherein the identified inferencing server is caused to load only a portion of the target model to fulfilling the inferencing request.

### Claim 19 (depends on 11)

The computer program product of claim 11, the program instructions readable and/or executable by the computer to cause the computer to: obtain, by the computer, an inference generated from the identified inferencing server, and store, by the computer, the inference to a predetermined table of a hardware memory module.

### Claim 20

A system, comprising:
a processor; and
logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to:
determine a plurality of downstream task models of a foundation model;
arrange the downstream task models into a logical tree structure, wherein each node of the logical tree structure represents a sequence of layers of an associated one of the downstream task models;
in response to a determination that a request for inferencing on a target model has resulted in a cache miss occurring, use the logical tree structure to identify an inferencing server that satisfies at least a first predetermined prerequisite for fulfilling the inferencing request; and
cause the identified inferencing server to fulfill the inferencing request.
