# US 2023/0144662 A1 · Techniques for partitioning neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0144662 A1
- Document ID: US20230144662A1
- Status: Published application
- Kind: A1
- Application number: 17/522,562
- Filed: 2021-11-09
- Published: 2023-05-11
- Earliest claimed: 2021-11-09
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Penn Tasinga; David B. Yastremsky
- Web page: https://patentgrep.com/patent/US20230144662A1

## CPC

- G06F 9/5022: {Mechanisms to release resources}
- G06F 9/5088: {involving task migration}
- G06F 9/505: {considering the load}
- G06F 9/5066: {Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451)}
- G06F 9/5077: {Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455)}
- G06F 9/5094: {where the allocation takes into account power or heat criteria (power management in computers in general G06F1/3203; thermal management in computers in general G06F1/206)}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks
- G06N 5/04: Inference or reasoning models
- G06F 2209/509: Offload
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Apparatuses, systems, and techniques to partition neural networks. In at least one embodiment, one or more circuits are to cause one or more neural networks to be dynamically partitioned based, at least in part, on one or more performance metrics of the one or more neural networks.

## Claims

### Claim 1

A processor, comprising:
one or more circuits to cause one or more neural networks to be dynamically partitioned based, at least in part, on one or more performance metrics of the one or more neural networks.

### Claim 2 (depends on 1)

The processor of claim 1, wherein the one or more performance metrics include an inferencing request metric.

### Claim 3 (depends on 1)

The processor of claim 1, wherein the one or more circuits are to cause the one or more neural networks to be dynamically partitioned on a plurality of graphics processing units (GPUs).

### Claim 4 (depends on 1)

The processor of claim 1, wherein the one or more circuits are to cause the one or more neural networks to be dynamically partitioned on a first one or more graphics processing units (GPUs) of a first computer system, and a second one or more GPUs of a second computer system.

### Claim 5 (depends on 1)

The processor of claim 1, wherein the one or more circuits are to generate one or more virtual representations of a corresponding one or more of the dynamically partitioned one or more neural networks.

### Claim 6 (depends on 1)

The processor of claim 1, wherein the one or more circuits are to allocate the dynamically partitioned one or more neural networks on one or more inference nodes.

### Claim 7 (depends on 1)

The processor of claim 1, wherein the one or more performance metrics include one or more performance metrics of one or more graphics processing units (GPUs).

### Claim 8 (depends on 1)

The processor of claim 1, wherein the one or more performance metrics include one or more inferencing request metrics, and the one or more circuits are to cause the one or more neural networks to be dynamically partitioned also based, at least in part, on one or more graphics processing unit metrics.

### Claim 9

A system, comprising:
one or more processors to cause one or more neural networks to be dynamically partitioned based, at least in part, on one or more performance metrics of the one or more neural networks; and
one or more memories to store one or more of the one or more performance metrics.

### Claim 10 (depends on 9)

The system of claim 9, wherein the one or more performance metrics include an inferencing request throughput or an inferencing request latency.

### Claim 11 (depends on 9)

The system of claim 9, wherein the one or more processors are to also to cause the one or more neural networks to be dynamically partitioned based, at least in part, on one or more memory metrics.

### Claim 12 (depends on 9)

The system of claim 9, wherein requests to use the partitioned one or more neural networks are to be routed via a corresponding one or more non-partitioned virtual neural network models.

### Claim 13 (depends on 9)

The system of claim 9, wherein the one or more processors are to allocate the dynamically partitioned one or more neural networks on two or more inference nodes.

### Claim 14 (depends on 9)

The system of claim 9, wherein the one or more processors are included in an inferencing system that is to receive inferencing requests via a network.

### Claim 15 (depends on 9)

The system of claim 9, wherein the one or more processors are to cause the one or more neural networks to be dynamically partitioned based, at least in part, on one or more requests to perform operations using the one or more neural networks.

### Claim 16

A method, comprising:
dynamically partitioning one or more neural networks based, at least in part, on one or more performance metrics of the one or more neural networks.

### Claim 17 (depends on 16)

The method of claim 16, wherein the one or more performance metrics include one or more inferencing request performance metrics.

### Claim 18 (depends on 16)

The method of claim 16, wherein dynamically partitioning the one or more neural networks includes partitioning the one or more neural networks in response to a first inferencing request, and repartitioning the one or more neural networks based, at least in part, on the one or more performance metrics in response to a second inferencing request.

### Claim 19 (depends on 16)

The method of claim 16, wherein dynamically partitioning the one or more neural networks is also performed based, at least in part, on one or more graphics processing unit (GPU) power metrics.

### Claim 20 (depends on 16)

The method of claim 16, wherein the one or more neural networks are a first one or more neural networks, and dynamically partitioning the one or more neural networks is based, at least in part, on a second one or more neural networks that use the one or more performance metrics as one or more inputs.

### Claim 21 (depends on 16)

The method of claim 16, wherein dynamically partitioning includes partitioning a first model that includes a first one or more neural networks, and partitioning a second model that includes a second one or more neural networks, and wherein a first partition of the first model is to be processed using a graphics processing unit (GPU), and a second partition of the second model is to be processed using the GPU.

### Claim 22

A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, is to cause the one or more processors to at least:
dynamically partition one or more neural networks based, at least in part, on one or more performance metrics of the one or more neural networks.

### Claim 23 (depends on 22)

The machine-readable medium of claim 22, wherein the one or more performance metrics are based, at least in part, on one or more requests to perform inferencing operations received over a network.

### Claim 24 (depends on 22)

The machine-readable medium of claim 22, wherein the one or more performance metrics include a throughput metric or a latency metric.

### Claim 25 (depends on 22)

The machine-readable medium of claim 22, wherein the one or more performance metrics include one or more inferencing request performance metrics, and the set of instructions which if performed by the one or more processors is also to cause the one or more processors to dynamically partition the one or more neural networks based, at least in part, on a set of available computing devices.

### Claim 26 (depends on 22)

The machine-readable medium of claim 22, wherein the one or more performance metrics include a metric based, at least in part, on power consumption.

### Claim 27 (depends on 22)

The machine-readable medium of claim 22, wherein the set of instructions, which if performed by the one or more processors, is to generate one or more non-partitioned virtual representations of a corresponding one or more of the dynamically partitioned one or more neural networks.

### Claim 28 (depends on 22)

The machine-readable medium of claim 22, wherein the one or more performance metrics are based, at least in part, on one or more medical image inferencing requests.
