# US 2021/0081806 A1 · Using a runtime engine to facilitate dynamic adaptation of deep neural networks for efficient processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081806 A1
- Document ID: US20210081806A1
- Status: Published application
- Kind: A1
- Application number: 17/016,889
- Filed: 2020-09-10
- Published: 2021-03-18
- Earliest claimed: 2019-09-13
- Assignees: Latent AI, Inc.
- Applicants: Latent AI, Inc.
- Inventors: Sek Meng Chai; Jagadeesh Kandasamy
- Web page: https://patentgrep.com/patent/US20210081806A1

## CPC

- G06N 3/08: Learning methods
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/094: Adversarial learning
- G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks
- G06N 5/04: Inference or reasoning models
- H04L 41/0816: the condition being an adaptation, e.g. in response to network events
- H04L 41/145: {involving simulating, designing, planning or modelling of a network}
- H04L 41/16: using machine learning or artificial intelligence
- G06N 3/047: Probabilistic or stochastic networks
- H04L 41/142: using statistical or mathematical methods
- H04L 43/045: for graphical visualisation of monitoring data
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

The disclosed embodiments relate to a system that facilitates dynamic runtime execution of a deep neural network (DNN). During operation, the system receives a model, a set of weights and runtime metadata for the DNN. The system also obtains code to perform inference-processing operations for the DNN. Next, the system compiles code to implement a runtime engine that facilitates throttling operations during execution of the inference-processing operations, wherein the runtime engine conserves computing resources by selecting portions of the inference-processing operations to execute based on the runtime metadata.

## Claims

### Claim 1

A method for facilitating dynamic runtime execution of a deep neural network (DNN), comprising:
receiving a model, a set of weights and runtime metadata for the DNN;
obtaining code to perform inference-processing operations for the DNN; and
compiling code to implement a runtime engine that facilitates throttling operations during execution of the inference-processing operations, wherein the runtime engine conserves computing resources by selecting portions of the inference-processing operations to execute based on the runtime metadata.

### Claim 2 (depends on 1)

The method of claim 1, wherein during the throttling operations, the runtime engine identifies portions of the DNN to mask out and/or selects portions of the DNN to traverse based on a graph analysis involving the model, the set of weights and the runtime metadata.

### Claim 3 (depends on 2)

The method of claim 2, wherein the graph analysis involves scheduling data-fetching operations based on the runtime metadata to facilitate execution of the DNN.

### Claim 4 (depends on 1)

The method of claim 1, wherein the runtime metadata comprises information about statistically relevant execution paths through the DNN, which are determined based on activations in the DNN and associated sub-tensors.

### Claim 5 (depends on 1)

The method of claim 1, wherein the runtime metadata specifies runtime masks to facilitate selectively executing inference-processing operations for the DNN.

### Claim 6 (depends on 5)

The method of claim 5, wherein the runtime metadata specifies different masks for different DNN outputs.

### Claim 7 (depends on 1)

The method of claim 1, wherein the runtime engine is configured to cryptographically decode a watermark pattern encoded in the set of weights to facilitate validating the DNN.

### Claim 8 (depends on 1)

The method of claim 1, wherein the runtime engine is configured to determine a current runtime state and a target runtime state for the DNN, wherein the throttling operations select an operational plan to achieve the target runtime state from the current runtime state.

### Claim 9 (depends on 8)

The method of claim 8, wherein the current runtime state comprises a current execution context and a current input context, wherein the current execution context comprises current activations and/or outputs for the DNN, and the current input context comprises contextual features associated with current inputs to the DNN.

### Claim 10 (depends on 1)

The method of claim 1, wherein the runtime engine is configured to decode the set of weights during the inference-processing operations based on dictionary index values in the runtime metadata.

### Claim 11

A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for facilitating dynamic runtime execution of a deep neural network (DNN), the method comprising:
receiving a model, a set of weights and runtime metadata for the DNN;
obtaining code to perform inference-processing operations for the DNN; and
compiling code to implement a runtime engine that facilitates throttling operations during execution of the inference-processing operations, wherein the runtime engine conserves computing resources by selecting portions of the inference-processing operations to execute based on the runtime metadata.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein during the throttling operations, the runtime engine identifies portions of the DNN to mask out and/or selects portions of the DNN to traverse based on a graph analysis involving the model, the set of weights and the runtime metadata.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein the graph analysis involves scheduling data-fetching operations based on the runtime metadata to facilitate execution of the DNN.

### Claim 14 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the runtime metadata comprises information about statistically relevant execution paths through the DNN, which are determined based on activations in the DNN and associated sub-tensors.

### Claim 15 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the runtime metadata specifies runtime masks to facilitate selectively executing inference-processing operations for the DNN.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the runtime metadata specifies different masks for different DNN outputs.

### Claim 17 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the runtime engine is configured to cryptographically decode a watermark pattern encoded in the set of weights to facilitate validating the DNN.

### Claim 18 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the runtime engine is configured to determine a current runtime state and a target runtime state for the DNN, wherein the throttling operations select an operational plan to achieve the target runtime state from the current runtime state.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein the current runtime state comprises a current execution context and a current input context, wherein the current execution context comprises current activations and/or outputs for the DNN, and the current input context comprises contextual features associated with current inputs to the DNN.

### Claim 20 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the runtime engine is configured to decode the set of weights during the inference-processing operations based on dictionary index values in the runtime metadata.

### Claim 21

A system that facilitates dynamic runtime execution of a deep neural network (DNN), comprising:
at least one processor and at least one associated memory; and
a processing mechanism that executes on the at least one processor, wherein during operation, the processing mechanism:
receives a model, a set of weights and runtime metadata for the DNN;
obtains code to perform inference-processing operations for the DNN; and
compiles code to implement a runtime engine that facilitates throttling operations during execution of the inference-processing operations, wherein the runtime engine conserves computing resources by selecting portions of the inference-processing operations to execute based on the runtime metadata.

### Claim 22 (depends on 21)

The system of claim 21, wherein during the throttling operations, the runtime engine identifies portions of the DNN to mask out and/or selects portions of the DNN to traverse based on a graph analysis involving the model, the set of weights and the runtime metadata.
