# US 2021/0081789 A1 · Optimizing execution of a neural network based on operational performance parameters

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081789 A1
- Document ID: US20210081789A1
- Status: Published application
- Kind: A1
- Application number: 17/016,908
- Filed: 2020-09-10
- Published: 2021-03-18
- Earliest claimed: 2019-09-13
- Assignees: Latent AI, Inc.
- Applicants: Latent AI, Inc.
- Inventors: Sek Meng Chai; Jagadeesh Kandasamy
- Web page: https://patentgrep.com/patent/US20210081789A1

## CPC

- G06N 3/08: Learning methods
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/094: Adversarial learning
- G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks
- G06N 5/04: Inference or reasoning models
- H04L 41/0816: the condition being an adaptation, e.g. in response to network events
- H04L 41/145: {involving simulating, designing, planning or modelling of a network}
- H04L 41/16: using machine learning or artificial intelligence
- G06N 3/047: Probabilistic or stochastic networks
- H04L 41/142: using statistical or mathematical methods
- H04L 43/045: for graphical visualisation of monitoring data
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

The disclosed embodiments relate to a system that optimizes execution of a DNN based on operational performance parameters. During operation, the system collects the operational performance parameters from the DNN during operation of the DNN, wherein the operational performance parameters include parameters associated with operating conditions for the DNN, parameters associated with resource utilization during operation of the DNN, and parameters associated with accuracy of results produced by the DNN. Next, the system uses the operational performance parameters to update the DNN model to improve performance and efficiency during execution of the DNN.

## Claims

### Claim 1

A method for optimizing execution of a deep neural network (DNN) based on operational performance parameters, comprising:
collecting the operational performance parameters from the DNN during operation of the DNN;
wherein the operational performance parameters include parameters associated with operating conditions for the DNN, parameters associated with resource utilization during operation of the DNN, and parameters associated with accuracy of results produced by the DNN; and
using the operational performance parameters to update the DNN model to improve performance and efficiency during execution of the DNN.

### Claim 2 (depends on 1)

The method of claim 1, wherein the method further comprises deploying and executing the updated DNN model at a location in a hierarchy of computing nodes, wherein the location is determined based on a global system-level optimization.

### Claim 3 (depends on 2)

The method of claim 2, wherein the operational performance parameters include information that is used to optimize overall network bandwidth within the hierarchy of computing nodes in which the DNN operates.

### Claim 4 (depends on 1)

The method of claim 1, wherein the operational performance parameters include profiling data, which identifies pathways within the DNN that are activated while the DNN performs inference-processing operations.

### Claim 5 (depends on 4)

The method of claim 4, wherein the profiling data is used to synthesize additional training data, which is used to train the updated DNN model to improve robustness.

### Claim 6 (depends on 4)

The method of claim 4, wherein while executing the DNN, a runtime engine for the DNN selectively activates pathways in the DNN to facilitate computationally efficient inference-processing operations.

### Claim 7 (depends on 1)

The method of claim 1, wherein the operational performance parameters are analyzed to determine coefficients for regularizer terms in a loss function that is used to train the updated DNN model, wherein the regularizer terms include a quantization term, which represents differences between pre-quantization and post-quantization weight values in the DNN, and a magnitude term, which represents magnitudes of the weight values.

### Claim 8 (depends on 1)

The method of claim 1, wherein a runtime engine for the DNN uses a policy generated using the operational performance parameters to achieve two or more of the following objectives:
maximizing classification accuracy of the DNN;
minimizing computational operations performed while executing the DNN;
minimizing power consumption of a device, which is executing the DNN; and
minimizing latency involved in executing the DNN to produce an output.

### Claim 9 (depends on 1)

The method of claim 1, wherein the updated DNN model is comprised of a plurality of DNN models trained simultaneously based on the operational performance parameters.

### Claim 10

A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for optimizing execution of a deep neural network (DNN) based on operational performance parameters, the method comprising:
collecting the operational performance parameters from the DNN during operation of the DNN;
wherein the operational performance parameters include parameters associated with operating conditions for the DNN, parameters associated with resource utilization during operation of the DNN, and parameters associated with accuracy of results produced by the DNN; and
using the operational performance parameters to update the DNN model to improve performance and efficiency during execution of the DNN.

### Claim 11 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the method further comprises deploying and executing the updated DNN model at a location in a hierarchy of computing nodes, wherein the location is determined based on a global system-level optimization.

### Claim 12 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the operational performance parameters include information that is used to optimize overall network bandwidth within the hierarchy of computing nodes in which the DNN operates.

### Claim 13 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the operational performance parameters include profiling data, which identifies pathways within the DNN that are activated while the DNN performs inference-processing operations.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the profiling data is used to synthesize additional training data, which is used to train the updated DNN model to improve robustness.

### Claim 15 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein while executing the DNN, a runtime engine for the DNN selectively activates pathways in the DNN to facilitate computationally efficient inference-processing operations.

### Claim 16 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the operational performance parameters are analyzed to determine coefficients for regularizer terms in a loss function that is used to train the updated DNN model, wherein the regularizer terms include a quantization term, which represents differences between pre-quantization and post-quantization weight values in the DNN, and a magnitude term, which represents magnitudes of the weight values.

### Claim 17 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein a runtime engine for the DNN uses a policy generated using the operational performance parameters to achieve two or more of the following objectives:
maximizing classification accuracy of the DNN;
minimizing computational operations performed while executing the DNN;
minimizing power consumption of a device, which is executing the DNN; and
minimizing latency involved in executing the DNN to produce an output.

### Claim 18 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the updated DNN model is comprised of a plurality of DNN models trained simultaneously based on the operational performance parameters.

### Claim 19

A system that optimizes execution of a deep neural network (DNN) based on operational performance parameters, comprising:
at least one processor and at least one associated memory; and
a processing mechanism that executes on the at least one processor, wherein during operation, the processing mechanism:
collects the operational performance parameters from the DNN during operation of the DNN, wherein the operational performance parameters include parameters associated with operating conditions for the DNN, parameters associated with resource utilization during operation of the DNN, and parameters associated with accuracy of results produced by the DNN; and
uses the operational performance parameters to update the DNN model to improve performance and efficiency during execution of the DNN.

### Claim 20 (depends on 19)

The system of claim 19, wherein the processing mechanism also deploys and executes the updated DNN model at a location in a hierarchy of computing nodes, wherein the location is determined based on a global system-level optimization.
