# US 12,602,921 B2 · Techniques for modifying and training a neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,921 B2
- Document ID: US12602921B2
- Status: Granted
- Kind: B2
- Application number: 16/885,170
- Filed: 2020-05-27
- Published: 2021-12-02
- Granted: 2026-04-14
- Earliest claimed: 2020-05-27
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Wentao Zhu; Daguang Xu; Can Zhao; Ziyue Xu; Holger Reinhard Roth
- Web page: https://patentgrep.com/patent/US12602921B2

## CPC

- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- G06F 9/3877: {using a secondary processor, e.g. coprocessor (peripheral processor G06F13/12)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 20/00: Machine learning
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Apparatuses, systems, and techniques are described herein to speed up inferencing in a neural network by copying output from one layer of the neural network to another computing resource based on dependencies among layers in the network. In at least one embodiment, a processor comprising one or more circuits causes two or more subsequent layers of one or more neural networks to be performed on separate computing resources from a previous layer of the one or more neural networks.

## Claims

### Claim 1

A processor, comprising:
one or more circuits to cause two or more subsequent layers of one or more neural networks to be performed on separate computing resources from a precedent layer of the one or more neural networks based, at least in part, on whether a same output of the precedent layer is to be provided to both the two or more subsequent layers, wherein the same output of the precedent layer is to be provided to both the two or more subsequent layers by:
determining that both the two or more subsequent layers are dependent on the same output from precedent layer;
copying the same output of the precedent layer; and
providing the same output of the precedent layer to a first subsequent layer of the two or more subsequent layers and a copied output to a second subsequent layer of the two or more subsequent layers.

### Claim 2 (depends on 1)

The processor of claim 1, further comprising:
the one or more circuits to:
modify the one or more neural networks to reduce an amount of dependencies between the precedent layer and the two or more subsequent layers of the one or more neural networks; and
partition the modified one or more neural networks to be trained using pipeline parallelism by:
copying data from a component of the one or more neural networks from a computing resource to another computing resource, wherein the computing resource from which data is being copied performs the component of the one or more neural networks dependent on the data and the another computing resource to which the data is copied performs another component of the one or more neural networks dependent on the data.

### Claim 3 (depends on 1)

The processor of claim 1, wherein both the two or more subsequent layers comprises at least an encoder in the first subsequent layer and a decoder in the second subsequent layer.

### Claim 4 (depends on 3)

The processor of claim 3, wherein the same output is provided to the encoder in the first subsequent layer and the copied output is provided to the decoder in the second subsequent layer.

### Claim 5 (depends on 4)

The processor of claim 4, wherein the first subsequent layer is to be performed by the processor and the second subsequent layer is to be performed by a different processor.

### Claim 6 (depends on 1)

The processor of claim 1, further comprising:
the one or more circuits to cause both the two or more subsequent layers of the one or more neural networks to be performed, at a same time, using the same output of the precedent layer.

### Claim 7 (depends on 1)

The processor of claim 1, wherein the separate computing resources comprises:
graphical processing units (GPUs), parallel processing units (PPUs), computer systems, pipelines, multiprocessors, other circuits of a different processor, and/or other circuits of a same processor.

### Claim 8

A system, comprising:
one or more computers having one or more processors to cause two or more subsequent layers of one or more neural networks to be performed on separate computing resources from a previous layer of the one or more neural networks based, at least in part, on whether a same output of the previous layer is to be provided to both the two or more subsequent layers, wherein the same output of the previous layer is to be provided to both the two or more subsequent layers by:
determining that both the two or more subsequent layers are dependent on output from the previous layer;
copying the output of the previous layer; and
providing the output of the previous layer to a first subsequent layer of the two or more subsequent layers and a copied output to a second subsequent layer of the two or more subsequent layers.

### Claim 9 (depends on 8)

The system of claim 8, wherein the previous layer comprises a layer of the one or more neural networks that is performed prior to both the two or more subsequent layers.

### Claim 10 (depends on 8)

The system of claim 8, wherein the same output of the previous layer is to be provided to both the two or more subsequent layers further comprises:
generating a duplicated copy of the output of the previous layer; and
providing the duplicated copy to one of the two or more subsequent layers.

### Claim 11 (depends on 8)

The system of claim 8, wherein the previous layer comprises an encoder and both the two or more subsequent layers comprise a second encoder in a first subsequent layer and a decoder in a second subsequent layer.

### Claim 12 (depends on 8)

The system of claim 8, further comprising the one or more computers having one or more processors to modify a parallel block from the one or more neural networks to remove a dependency within the parallel block.

### Claim 13 (depends on 8)

The system of claim 8, further comprising the one or more computers having one or more processors to train the one or more neural networks by reducing dependencies of the output from the previous layer with the two or more subsequent layers.

### Claim 14 (depends on 8)

The system of claim 8, further comprising the one or more computers having one or more processors to train the one or more neural networks to perform medical image segmentation.

### Claim 15 (depends on 8)

The system of claim 8, further comprise the one or more computers having one or more processors to partition the one or more neural networks to be trained using pipeline parallelism.

### Claim 16

A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to cause two or more subsequent layers of one or more neural networks to be performed on separate computing resources from a preceding layer of the one or more neural networks based, at least in part, on whether a same output of the preceding layer is to be provided to both the two or more subsequent layers, wherein the same output of the preceding layer is to be provided to both the two or more subsequent layers by:
determining that both the two or more subsequent layers are dependent on output from preceding layer;
copying the output of the preceding layer; and
providing the output of the preceding layer to a first subsequent layer of the two or more subsequent layers and a copied output to a second subsequent layer of the two or more subsequent layers.

### Claim 17 (depends on 16)

The non-transitory machine-readable medium of claim 16, wherein the set of instructions, which if performed by the one or more processors, further cause the one or more processors to:
modify the one or more neural networks by modifying a first parallel block to reduce an amount of dependencies between layers of the first parallel block; and
partition the modified one or more neural networks to be trained using pipeline parallelism.

### Claim 18 (depends on 17)

The non-transitory machine-readable medium of claim 17, wherein the set of instructions, which if performed by the one or more processors, further cause the one or more processors to generate a second parallel block, wherein layers in the second parallel block is to be performed by a different computing resource than layers in the first parallel block.

### Claim 19 (depends on 18)

The non-transitory machine-readable medium of claim 18, wherein the set of instructions, which if performed by the one or more processors, further cause the one or more processors to copy data from a component of the one or more neural networks from the first parallel block to the second parallel block such that the different computing resource performs another component of the one or more neural networks using the copied data.

### Claim 20 (depends on 19)

The non-transitory machine-readable medium of claim 19, wherein the output of the preceding layer from the first parallel block is provided to a second encoder in a first subsequent layer of the two or more subsequent layers and a copy of the output is provided to a decoder in the second parallel block.

### Claim 21 (depends on 16)

The non-transitory machine-readable medium of claim 16, wherein the set of instructions, which if performed by the one or more processors, further cause the one or more processors to cause both the two or more subsequent layers of the one or more neural networks to be performed, simultaneously, using the same output of the preceding layer.

### Claim 22 (depends on 16)

The non-transitory machine-readable medium of claim 16, wherein the set of instructions, which if performed by the one or more processors, further cause the one or more processors to train the one or more neural networks by using a combination of training data parallelism and model parallelism.