# US 2021/0334644 A1 · Neural network training technique

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334644 A1
- Document ID: US20210334644A1
- Status: Published application
- Kind: A1
- Application number: 16/859,360
- Filed: 2020-04-27
- Published: 2021-10-28
- Earliest claimed: 2020-04-27
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Zhiding Yu; Wuyang Chen; Anima Anandkumar
- Web page: https://patentgrep.com/patent/US20210334644A1

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/08: Learning methods
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/096: Transfer learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Apparatuses, systems, and techniques to train one or more neural networks. In at least one embodiment, one or more neural networks are trained based, at least in part, on inferencing output from one or more second neural networks.

## Claims

### Claim 1

A processor, comprising:
one or more circuits to train one or more first neural networks based, at least in part, on inferencing output from a one or more second neural networks trained using non-synthetic images.

### Claim 2 (depends on 1)

The processor of claim 1, wherein a learning rate of the one or more first neural networks is adjusted, during training, based at least in part on output of the one or more second neural networks.

### Claim 3 (depends on 2)

The processor of claim 2, wherein the learning rate adjusted for a region of the one or more first neural networks, the region comprising layers grouped based, at least in part, on input resolution.

### Claim 4 (depends on 1)

The processor of claim 1, the one or more circuits to adjust training of the one or more first neural networks based, at least in part, on output of the one or more second neural networks in response to an input based on synthetic image data.

### Claim 5 (depends on 4)

The processor of claim 4, the one or more circuits to compute a learning rate scaling factor and adjust a learning rate of the one or more first neural networks based, at least in part, on the learning rate scaling factor.

### Claim 6 (depends on 1)

The processor of claim 1, wherein the one or more second neural networks are trained to perform an image processing task equivalent to an image processing task performed by the one or more first neural networks.

### Claim 7 (depends on 1)

The processor of claim 1, wherein parameters of the one or more second neural networks are not adjusted based on input derived from synthetic images.

### Claim 8 (depends on 1)

The processor of claim 1, wherein the first one or more neural networks and the second one or more neural networks have equivalent structures.

### Claim 9

A system, comprising:
one or more processors to train one or more first neural networks based, at least in part, on inferencing output from a one or more second neural networks trained using non-synthetic images.

### Claim 10 (depends on 9)

The system of claim 9, wherein a learning rate of the one or more first neural networks is adjusted based, at least in part, on output of the one or more second neural networks.

### Claim 11 (depends on 10)

The system of claim 10, the one or more processors to adjust the learning rate for a region of the one or more first neural networks.

### Claim 12 (depends on 11)

The system of claim 11, wherein the region comprises layers grouped according to input resolution.

### Claim 13 (depends on 9)

The system of claim 9, the one or more processors to adjust training of the one or more first neural networks based, at least in part, on output of the one or more second neural networks in response to an input based on synthetic image data.

### Claim 14 (depends on 9)

The system of claim 9, the one or more processors to compute a learning rate scaling factor based, at least in part, on a divergence factor, the divergence factor computed based, at least in part, on the output of the one or more second neural networks.

### Claim 15 (depends on 9)

The system of claim 9, wherein the one or more second neural networks are frozen during training of the one or more first neural networks.

### Claim 16

A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
train one or more first neural networks based, at least in part, on inferencing output from a one or more second neural networks trained using non-synthetic images.

### Claim 17 (depends on 16)

The machine-readable medium of claim 16, having stored thereon a set of instructions which if performed by one or more processors, cause the one or more processors to at least:
compute an adjustment to a learning rate of the one or more first neural networks, based at least in part on output of the one or more second neural networks.

### Claim 18 (depends on 17)

The machine-readable medium of claim 17, wherein the adjustment to the learning rate is calculated for a region of the one or more first neural networks, the region comprising layers grouped based, at least in part, on input resolution.

### Claim 19 (depends on 18)

The machine-readable medium of claim 18, wherein the adjustment to the learning rate is calculated based, at least in part, by a long short-term memory (“LSTM”) module.

### Claim 20 (depends on 16)

The machine-readable medium of claim 16, wherein the output of the one or more second neural networks is based, at least in part, on input based on synthetic image data.

### Claim 21 (depends on 16)

The machine-readable medium of claim 16, wherein the one or more second neural networks are trained to perform an image processing task equivalent to an image processing task performed by the one or more first neural networks.

### Claim 22 (depends on 16)

The machine-readable medium of claim 16, wherein parameters of the one or more second neural networks are not adjusted based on input derived from synthetic images.

### Claim 23 (depends on 16)

The machine-readable medium of claim 16, wherein the one or more first neural networks, once trained, are usable to perform an inferencing task independently of the one or more second neural networks.

### Claim 24

A computing device, comprising:
one or more processors to perform an image processing task based at least in part on one or more first neural networks trained based, at least in part, on inferencing output from a one or more second neural networks trained using non-synthetic images.

### Claim 25 (depends on 24)

The computing device of claim 24, wherein the image processing task comprises at least one of recognition or classification.

### Claim 26 (depends on 24)

The computing device of claim 24, the one or more processors to compute an adjustment to a learning rate of the one or more first neural networks, based at least in part on output of the one or more second neural networks.

### Claim 27 (depends on 26)

The computing device of claim 26, wherein the adjustment to the learning rate is calculated for a region of the one or more first neural networks, the region comprising layers grouped based, at least in part, on input resolution.

### Claim 28 (depends on 24)

The computing device of claim 24, wherein the output of the one or more second neural networks is based, at least in part, on input based on synthetic image data.

### Claim 29 (depends on 24)

The computing device of claim 24, wherein the one or more first neural networks and one or more second neural networks are each trained to perform the image processing task.

### Claim 30 (depends on 29)

The computing device of claim 29, wherein the one or more second neural networks are trained only on real images.
