# US 2024/0177026 A1 · System and method for managing inference model performance through inference generation path restructuring

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0177026 A1
- Document ID: US20240177026A1
- Status: Published application
- Kind: A1
- Application number: 18/060,112
- Filed: 2022-11-30
- Published: 2024-05-30
- Earliest claimed: 2022-11-30
- Assignees: Dell Products L.P.
- Applicants: Dell Products L.P.
- Inventors: OFIR EZRIELEV; JEHUDA SHEMER; TOMER KUSHNIR
- Web page: https://patentgrep.com/patent/US20240177026A1

## CPC

- G06N 5/043: Distributed expert systems; Blackboards
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods and systems for managing execution of an inference model hosted by data processing systems are disclosed. To manage execution of the inference model hosted by the data processing systems, a system may include an inference model manager and any number of data processing systems. The inference model manager may monitor the risk of unsuccessful execution of the inference model by the data processing systems and may proactively take action to support inference generation in the event of reduced functionality of one or more of the data processing systems. The inference model manager may distribute multiple redundant instances of the inference model so that each data processing system only hosts one instance of the inference model. Inference model manager may also obtain an execution plan to respond to a failure of one or more data processing systems to ensure no inference model bottlenecks occur during re-deployment of the inference model.

## Claims

### Claim 1

A method of managing execution of an inference model hosted by data processing systems, the method comprising:
identifying that a first data processing system of the data processing systems has a level of risk of failing to execute a portion of the inference model that is above a threshold;
based on the identification:
performing an inference generation path analysis for the first data processing system to identify whether the first data processing system is an inference model bottleneck;
in an instance of the inference generation path analysis where the first data processing system is the inference model bottleneck:
obtaining a deployment plan that distributes multiple redundant instances of the inference model so that only one of the instances of the inference model is hosted by the first data processing system;
obtaining an execution plan for responding to a failure of the first data processing system, the execution plan ensuring that one or more other inference model bottlenecks are not formed when the only one of the instances of the inference model is re-deployed across the data processing systems based on the execution plan;
deploying the inference model across the data processing systems based on the deployment plan;
automatically initiating re-deployment of the inference model in response to a failure of the first data processing system based on the execution plan to obtain a re-deployed inference model; and
generating, using the re-deployed inference model, an inference.

### Claim 2 (depends on 1)

The method of claim 1, wherein performing the inference generation path analysis comprises:
identifying one or more portions of the inference model hosted by the first data processing system;
making an identification of an inference generation path associated with each of the one or more portions of the inference model hosted by the first data processing system; and
in an instance of the identification where there is more than one inference generation path associated with the first data processing system:
identifying the first data processing system as the inference model bottleneck.

### Claim 3 (depends on 2)

The method of claim 2, wherein a failure of the inference model bottleneck prevents timely execution of one or more redundant instances of the inference model.

### Claim 4 (depends on 3)

The method of claim 3, wherein the failure of the inference model bottleneck prevents timely execution of all of the redundant instances of the inference model deployed across the data processing systems.

### Claim 5 (depends on 2)

The method of claim 2, wherein the inference generation path comprises:
a listing of instances of each of the portions of the inference model usable to generate an inference model result; and
an ordering of the listing of the instances.

### Claim 6 (depends on 5)

The method of claim 5, wherein obtaining the deployment plan comprises:
identifying the one or more portions of the inference model hosted by the first data processing system;
identifying a second data processing system, the second data processing system currently not hosting any portions of the inference model;
obtaining an updated inference generation path for one of the portions of the inference model hosted by the first data processing system based on the second data processing system; and
obtaining inference generation instructions for the data processing systems that are members of the updated inference generation path.

### Claim 7 (depends on 6)

The method of claim 6, wherein the inference generation instructions indicate a processing result transmission destination for each of the data processing system that are members of the updated inference generation path.

### Claim 8 (depends on 7)

The method of claim 7, wherein the deployment plan ensures that each data processing system of the data processing systems is part of only one inference generation path for the inference model.

### Claim 9 (depends on 8)

The method of claim 8, wherein deploying the inference model across the data processing systems based on the deployment plan comprises:
configuring the data processing systems that are members of the updated inference generation path to forward processing results based on the inference generation instructions.

### Claim 10 (depends on 9)

The method of claim 9, wherein the execution plan indicates a failover inference generation path for an instance of the inference model hosted by the second data processing system.

### Claim 11 (depends on 10)

The method of claim 10, wherein the failover inference generation path comprises:
an updated listing of the instances of each of the portions of the inference model usable to generate the inference model result, the updated listing indicating replacement of the first data processing system with a third data processing system responsive to failure of the first data processing system, and the third data processing system not hosting any portion of the inference model prior to the failure of the first data processing system.

### Claim 12 (depends on 11)

The method of claim 11, wherein automatically initiating re-deployment of the inference model comprises:
identifying the failure of the first data processing system;
identifying the failover inference generation path based on the execution plan; and
re-deploying the inference model based on the failover inference generation path.

### Claim 13 (depends on 12)

The method of claim 12, wherein re-deploying the inference model comprises:
deploying the portion of the inference model hosted by the first data processing system to the third data processing system; and
transmitting updated inference generation instructions to the data processing systems, the updated inference generation instructions being based, at least in part, on the failover inference generation path.

### Claim 14

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing execution of an inference model hosted by data processing systems, the operations comprising:
identifying that a first data processing system of the data processing systems has a level of risk of failing to execute a portion of the inference model that is above a threshold;
based on the identification:
performing an inference generation path analysis for the first data processing system to identify whether the first data processing system is an inference model bottleneck;
in an instance of the inference generation path analysis where the first data processing system is the inference model bottleneck:
obtaining a deployment plan that distributes multiple redundant instances of the inference model so that only one of the instances of the inference model is hosted by the first data processing system;
obtaining an execution plan for responding to a failure of the first data processing system, the execution plan ensuring that one or more other inference model bottlenecks are not formed when the only one of the instances of the inference model is re-deployed across the data processing systems based on the execution plan;
deploying the inference model across the data processing systems based on the deployment plan;
automatically initiating re-deployment of the inference model in response to a failure of the first data processing system based on the execution plan to obtain a re-deployed inference model; and
generating, using the re-deployed inference model, an inference.

### Claim 15 (depends on 14)

The non-transitory machine-readable medium of claim 14, wherein performing the inference generation path analysis comprises:
identifying one or more portions of the inference model hosted by the first data processing system;
making an identification of an inference generation path associated with each of the one or more portions of the inference model hosted by the first data processing system; and
in an instance of the identification where there is more than one inference generation path associated with the first data processing system:
identifying the first data processing system as the inference model bottleneck.

### Claim 16 (depends on 15)

The non-transitory machine-readable medium of claim 15, wherein a failure of the inference model bottleneck prevents timely execution of one or more redundant instances of the inference model.

### Claim 17 (depends on 16)

The non-transitory machine-readable medium of claim 16, wherein the failure of the inference model bottleneck prevents timely execution of all of the redundant instances of the inference model deployed across the data processing systems.

### Claim 18

A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing execution of an inference model hosted by data processing systems, the operations comprising:
identifying that a first data processing system of the data processing systems has a level of risk of failing to execute a portion of the inference model that is above a threshold;
based on the identification:
performing an inference generation path analysis for the first data processing system to identify whether the first data processing system is an inference model bottleneck;
in an instance of the inference generation path analysis where the first data processing system is the inference model bottleneck:
obtaining a deployment plan that distributes multiple redundant instances of the inference model so that only one of the instances of the inference model is hosted by the first data processing system;
obtaining an execution plan for responding to a failure of the first data processing system, the execution plan ensuring that one or more other inference model bottlenecks are not formed when the only one of the instances of the inference model is re-deployed across the data processing systems based on the execution plan;
deploying the inference model across the data processing systems based on the deployment plan;
automatically initiating re-deployment of the inference model in response to a failure of the first data processing system based on the execution plan to obtain a re-deployed inference model; and
generating, using the re-deployed inference model, an inference.

### Claim 19 (depends on 18)

The data processing system of claim 18, wherein performing the inference generation path analysis comprises:
identifying one or more portions of the inference model hosted by the first data processing system;
making an identification of an inference generation path associated with each of the one or more portions of the inference model hosted by the first data processing system; and
in an instance of the identification where there is more than one inference generation path associated with the first data processing system:
identifying the first data processing system as the inference model bottleneck.

### Claim 20 (depends on 19)

The data processing system of claim 19, wherein a failure of the inference model bottleneck prevents timely execution of one or more redundant instances of the inference model.
