# US 12,001,954 B2 · Method of performing learning of deep neural network and apparatus thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,954 B2
- Document ID: US12001954B2
- Status: Granted
- Kind: B2
- Application number: 16/281,737
- Filed: 2019-02-21
- Published: 2019-08-22
- Granted: 2024-06-04
- Earliest claimed: 2018-02-20
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Sungho Kang; Hyungdal Kwon; Cheon Lee; Yunjae Lim
- Web page: https://patentgrep.com/patent/US12001954B2

## CPC

- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06F 7/58: Random or pseudo-random number generators
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An encoding apparatus connected to a learning circuit processing learning of a deep neural network and configured to perform encoding for reconfiguring connection or disconnection of a plurality of edges in a layer of the deep neural network using an edge sequence generated based on a random number sequence and dropout information indicating a ratio between connected edges and disconnected edges of a plurality of edges included in a layer of the deep neural network.

## Claims

### Claim 1

An encoding apparatus comprising:
a memory storing a random number sequence generated by a random number generator; and
at least one processor configured to:
receive dropout information of a deep neural network, the dropout information indicating a ratio between connected edges and disconnected edges of a plurality of first edges included in a first layer of the deep neural network,
generate an edge sequence indicating connection or disconnection of a plurality of second edges included in a second layer of the deep neural network by modifying the random number sequence based on a comparison between the dropout information and a target ratio value, and
output the edge sequence such that the connection or the disconnection of the plurality of second edges included in the second layer of the deep neural network are reconfigured,
wherein the at least one processor is further configured to modify the random number sequence such that a ratio value of a number of bits having a value of 0 and 1 in an entirety of the deep neural network is the target ratio value.

### Claim 2 (depends on 1)

The encoding apparatus of claim 1, wherein the random number sequence is based on a clock signal of the random number generator.

### Claim 3 (depends on 2)

The encoding apparatus of claim 2, wherein a size of the random number sequence is based on a number of the plurality of second edges in the second layer of the deep neural network.

### Claim 4 (depends on 3)

The encoding apparatus of claim 3, wherein the size of the random number sequence is equal to a quantity of edges in the first layer of the deep neural network.

### Claim 5 (depends on 2)

The encoding apparatus of claim 2, wherein the dropout information, the random number sequence, and the edge sequence have a bit width formed of binary numbers.

### Claim 6 (depends on 5)

The encoding apparatus of claim 5, wherein the at least one processor is further configured to generate the edge sequence based on a pattern of bits in the random number sequence having a bit value of 0 and bits in the random number sequence having a bit value of 1, and a pattern of bits in the dropout information having a bit value indicating a connected edge and bits in the dropout information having a bit value indicating a disconnected edge.

### Claim 7 (depends on 2)

The encoding apparatus of claim 2, wherein the edge sequence is a basis for a dropout operation in the first layer of the deep neural network.

### Claim 8 (depends on 2)

The encoding apparatus of claim 2, wherein the at least one processor is further configured to obtain weights of the plurality of second edges in the second layer of the deep neural network, perform a pruning operation based on a result of comparing the weights with a preset threshold weight, and generate the edge sequence to indicate the connection or the disconnection of the plurality of second edges of the second layer of the deep neural network based on the pruning operation.

### Claim 9 (depends on 1)

The encoding apparatus of claim 1, wherein the at least one processor is further configured to:
select one of a plurality of types of input signals and output the selected signal;
receive an operation result from the deep neural network to determine whether overflow has occurred in the operation result; and
perform a dynamic fixed point operation of modifying an expressible range of information used in the deep neural network based on whether overflow has occurred.

### Claim 10 (depends on 1)

The encoding apparatus of claim 1, further comprising a learning circuit including the deep neural network,
wherein the learning circuit is configured to:
receive the edge sequence,
change a connection state of at least one second edge included in the second layer of the deep neural network based on the edge sequence, and
perform a learning operation using the first layer.

### Claim 11 (depends on 1)

The encoding apparatus of claim 1, wherein the at least one processor is further configured to:
generate a corrected random number sequence by correcting at least one random number value included in the random number sequence based on the comparison between the dropout information and the target ratio value, and
use the corrected random number sequence as the edge sequence.

### Claim 12

An encoding method of an encoding apparatus, the encoding method comprising:
storing a random number sequence generated by a random number generator;
receiving dropout information of a deep neural network, the dropout information indicating a ratio between connected edges and disconnected edges of a plurality of first edges included in a first layer of the deep neural network;
generating an edge sequence indicating connection or disconnection of a plurality of second edges included in a second layer of the deep neural network by modifying the random number sequence based on a comparison between the dropout information and a target ratio value; and
outputting the edge sequence such that the connection or the disconnection of the plurality of second edges included in the second layer of the deep neural network are reconfigured,
wherein the generating of the edge sequence comprises modifying the random number sequence such that a ratio value of a number of bits having a value of 0 and 1 in an entirely of the deep neural network is the target ratio value.

### Claim 13 (depends on 12)

The encoding method of claim 12, wherein the random number sequence is based on a clock signal of the random number generator.

### Claim 14 (depends on 13)

The encoding method of claim 13, wherein a size of the random number sequence is based on a number of the plurality of second edges in the second layer of the deep neural network.

### Claim 15 (depends on 14)

The encoding method of claim 14, wherein the size of the random number sequence is equal to a quantity of edges in the first layer of the deep neural network.

### Claim 16 (depends on 13)

The encoding method of claim 13, wherein the dropout information, the random number sequence, and the edge sequence have a bit width formed of binary numbers.

### Claim 17 (depends on 16)

The encoding method of claim 16, wherein the generating of the edge sequence comprises generating the edge sequence based on a pattern of bits in the random number sequence having a bit value of 0 and bits in the random number sequence having a bit value of 1, and a pattern of bits in the dropout information having a bit value indicating a connected edge and bits in the dropout information having a bit value indicating a disconnected edge.

### Claim 18 (depends on 13)

The encoding method of claim 13, wherein the edge sequence is a basis for a dropout operation in the first layer of the deep neural network.

### Claim 19 (depends on 13)

The encoding method of claim 13, further comprising:
obtaining weights of the plurality of second edges of the second layer of the deep neural network;
performing a pruning operation based on a result of comparing the weights of the plurality of second edges with a preset threshold weight; and
generating the edge sequence to indicate the connection or the disconnection of the plurality of second edges of the second layer of the deep neural network based on the pruning operation.

### Claim 20 (depends on 12)

The encoding method of claim 12, further comprising:
receiving an operation result from the deep neural network;
determining whether overflow has occurred in the operation result; and
performing a dynamic fixed point operation of modifying an expressible range of a value used in the deep neural network based on whether overflow has occurred.