# US 2023/0409936 A1 · Proxy systems and methods for multiprocessing architectures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0409936 A1
- Document ID: US20230409936A1
- Status: Published application
- Kind: A1
- Application number: 18/318,860
- Filed: 2023-05-17
- Published: 2023-12-21
- Earliest claimed: 2022-05-17
- Assignees: Kinara, Inc.
- Applicants: Kinara, Inc.
- Inventors: Lava Kumar Bokam; Sriduth Jayhari; Divya Vipin; Rajashekar Reddy Ereddy; Snigdha Alkanti; Venkateswara Rao Andole Mankali; Suresh Kumar Vennam; Mohammed Mujahid; Sreenivas Aerra Reddy
- Web page: https://patentgrep.com/patent/US20230409936A1

## CPC

- G06N 3/063: using electronic means
- G06N 5/046: Forward inferencing; Production systems
- G06F 9/50: Allocation of resources, e.g. of the central processing unit [CPU]
- G06F 9/505: {considering the load}
- G06N 3/045: Combinations of networks
- G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks
- G06F 2209/503: Resource availability
- G06F 2209/509: Offload
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Proxy systems and methods for multiprocessing architectures are described. One method includes receiving an inference request and a statistics request from a client computing system. The method may access a load state of each processing device in a subset of processing devices preloaded with the neural network model, and select a target processing device from the subset based on the load states. One aspect includes transmitting the inference request to the target processing device, and monitoring an execution of the inference request by the target processing device based on the neural network model. The method may receive an inference result generated by the target processing device after executing the inference request, and compute an average inference time for the inference request execution based on the monitoring. The method may transmit the inference result and the average inference time to the client computing system.

## Claims

### Claim 1

A method comprising:
receiving an inference request from a client computing system, the inference request comprising a model ID associated with a neural network model and an input tensor;
receiving a statistics request from the client computing system, the statistics request including the model ID and an average inference time request;
accessing a load state of each processing device in a subset of processing devices preloaded with the neural network model;
selecting a target processing device from the subset based on the load states;
transmitting the inference request to the target processing device;
monitoring an execution of the inference request by the target processing device based on the neural network model;
receiving an inference result generated by the target processing device after executing the inference request;
computing the average inference time for the inference request execution based on the monitoring; and
transmitting the inference result and the average inference time to the client computing system.

### Claim 2 (depends on 1)

The method of claim 1, wherein the inference result is an output tensor.

### Claim 3 (depends on 1)

The method of claim 1, wherein the neural network is a convolutional neural network or a neural network comprised of one or more linear algebra operators.

### Claim 4 (depends on 1)

The method of claim 1, wherein the inference request includes an input tensor.

### Claim 5 (depends on 4)

The method of claim 4, wherein the input tensor is an image generated by an image sensor.

### Claim 6 (depends on 1)

The method of claim 1, wherein the inference result is an output tensor.

### Claim 7 (depends on 1)

The method of claim 1, wherein the input tensor is an image generated by an image sensor.

### Claim 8 (depends on 1)

The method of claim 1, wherein the load state includes an endpoint execution queue associated with each processing device.

### Claim 9 (depends on 1)

The method of claim 1, further comprising selecting the target processing device based on a model occupancy for the selected model ID.

### Claim 10 (depends on 9)

The method of claim 9, wherein the model occupancy is stored in a device library.

### Claim 11

An apparatus comprising:
a proxy computing system;
a client computing system communicatively coupled to the proxy computing system; and
a set of processing devices preloaded with a neural network model and communicatively coupled to the proxy computing system, wherein:
the proxy computing system receives an inference request from the client computing system, the inference request comprising a model ID associated with the neural network model and an input tensor;
the proxy computing system receives a statistics request from the client computing system, the statistics request including the model ID and an average inference time request;
the proxy computing system accesses a load state of each processing device in the set of processing devices;
the proxy computing system selects a target processing device from the set based on the load states;
the proxy computing system transmits the inference request to the target processing device;
the proxy computing system monitors an execution of the inference request by the target processing device based on the neural network model;
the proxy computing system receives an inference result generated by the target processing device after executing the inference request;
the proxy computing system computes the average inference time for the inference request execution based on the monitoring; and
the proxy computing system transmits the inference result and the average inference time to the client computing system.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the inference result is an output tensor.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein the neural network is a convolutional neural network or a neural network comprised of one or more linear algebra operators.

### Claim 14 (depends on 11)

The apparatus of claim 11, wherein the inference request includes an input tensor.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the input tensor is an image generated by an image sensor.

### Claim 16 (depends on 11)

The apparatus of claim 11, wherein the inference result is an output tensor.

### Claim 17 (depends on 11)

The apparatus of claim 11, wherein the input tensor is an image generated by an image sensor.

### Claim 18 (depends on 11)

The apparatus of claim 11, wherein the load state includes an endpoint execution queue associated with each processing device.

### Claim 19 (depends on 11)

The apparatus of claim 11, wherein the proxy computing system selects the target processing device based on a model occupancy for the selected model ID.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the model occupancy is stored in a device library.
