# US 2021/0241112 A1 · Neural network update method, classification method and electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241112 A1
- Document ID: US20210241112A1
- Status: Published application
- Kind: A1
- Application number: 17/168,564
- Filed: 2021-02-05
- Published: 2021-08-05
- Earliest claimed: 2020-02-05
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Bin DAI; Zhouchen LIN
- Web page: https://patentgrep.com/patent/US20210241112A1

## CPC

- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06N 3/048: Activation functions
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A neural network update method, a classification method and an electronic device thereof are provided. The neural network update method includes obtaining input data of the neural network; identifying an active neuron and an inactive neuron from among at least one neuron included in the neural network; and based on an information bottleneck of the at least one neuron, training the neural network based on the input data while maintaining information bottleneck of the active neuron.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

An electronic apparatus comprising:
at least one memory configured to store a neural network comprising a plurality of network layers, each of the plurality of network layers including at least one neuron; and
at least one processor configured to:
obtain input data of the neural network,
identify an active neuron and an inactive neuron from among the at least one neuron included in the neural network,
based on an information bottleneck of the at least one neuron, train the neural network based on the input data while maintaining information bottleneck of the active neuron.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein the at least one processor is further configured to train the neural network by updating information bottleneck of the inactive neuron based on the input data while maintaining the information bottleneck of the active neuron.

### Claim 23 (depends on 21)

The apparatus of claim 21, wherein the at least one processor is further configured to:
based on the information bottleneck of the at least one neuron exceeding a predetermined value, identify the at least one neuron as the active neuron, and
based on the amount of information being less than or equal to the predetermined value, identify the at least one neuron as the inactive neuron.

### Claim 24 (depends on 21)

The apparatus of claim 21, wherein the plurality of network layer comprises an input layer, at least one hidden layer and an output layer.

### Claim 25 (depends on 24)

The apparatus of claim 24, wherein the at least one processor is further configured to:
maintain weights between the active neuron in one of the at least one hidden layer and the active neuron in a previous network layer unchanged during the training,
reset weights between the active neuron in the one of the at least one hidden layer and the inactive neuron in the previous network layer to 0, and maintain the reset weights unchanged during the training, and
train weights between the inactive neuron in the one of the at least one hidden layer and each neuron in the previous network layer during the training.

### Claim 26 (depends on 24)

The apparatus of claim 24, wherein the at least one processor is further configured to:
based on the output layer being a multi-head output layer:
maintain weights between the active neuron in the output layer and the active neuron in a previous network layer unchanged during the training,
reset weights between the active neuron in the output layer and the inactive neuron in the previous network layer to 0, and maintain the reset weights unchanged during the training,
train weights between the inactive neuron in the output layer and each neuron in the previous network layer during the training, and
based on the output layer being a single-head output layer:
train weights between each neuron in the output layer and each neuron in the previous network layer during the training.

### Claim 27 (depends on 24)

The apparatus of claim 24, wherein the at least one processor is further configured to:
add a neuron in one of the at least one hidden layer based on activation state of neuron included in the one of the at least one hidden layer, and
train the neural network based on the input data.

### Claim 28 (depends on 27)

The apparatus of claim 27, wherein the at least one processor is further configured to add the neuron in the one of the at least one hidden layer based on all neurons included in the one of the at least one hidden layer being the active neuron.

### Claim 29 (depends on 21)

The apparatus of claim 21, wherein the neural network is a neural network trained in a sparsity promoting manner which uses as few neurons as possible based on another input data.

### Claim 30

A control method of an electronic apparatus comprising:
obtaining input data of the neural network;
identifying an active neuron and an inactive neuron from among at least one neuron included in the neural network; and
based on an information bottleneck of the at least one neuron,
training the neural network based on the input data while maintaining information bottleneck of the active neuron.

### Claim 31 (depends on 30)

The control method of claim 30, wherein the training comprises training the neural network by updating information bottleneck of the inactive neuron based on the input data while maintaining the information bottleneck of the active neuron.

### Claim 32 (depends on 30)

The control method of claim 30, wherein the identifying comprises:
based on the information bottleneck of the at least one neuron exceeding a predetermined value, identifying the at least one neuron as the active neuron, and
based on the amount of information being less than or equal to the predetermined value, identifying the at least one neuron as the inactive neuron.

### Claim 33 (depends on 30)

The control method of claim 30, wherein the plurality of network layers comprises an input layer, at least one hidden layer and an output layer.

### Claim 34 (depends on 33)

The control method of claim 33, wherein the training comprises:
maintaining weights between the active neuron in one of the at least one hidden layer and the active neuron in a previous network layer unchanged during the training,
resetting weights between the active neuron in the one of the at least one hidden layer and the inactive neuron in the previous network layer to 0, and maintaining the reset weights unchanged during the training, and
training weights between the inactive neuron in the one of the at least one hidden layer and each neuron in the previous network layer during the training.

### Claim 35 (depends on 33)

The control method of claim 33, wherein the training comprises:
based on the output layer being a multi-head output layer:
maintaining weights between the active neuron in the output layer and the active neuron in a previous network layer unchanged during the training,
resetting weights between the active neuron in the output layer and the inactive neuron in the previous network layer to 0, and maintaining the reset weights unchanged during the training,
training weights between the inactive neuron in the output layer and each neuron in the previous network layer during the training, and
based on the output layer being a single-head output layer:
training weights between each neuron in the output layer and each neuron in the previous network layer during the training.

### Claim 36 (depends on 33)

The control method of claim 33, further comprising:
adding a neuron in one of the at least one hidden layer based on activation state of neuron included in the one of the at least one hidden layer.

### Claim 37 (depends on 36)

The control method of claim 36, wherein the adding the neuron comprises adding the neuron in the one of the at least one hidden layer based on all neurons included in the one of the at least one hidden layer being the active neuron.

### Claim 38 (depends on 30)

The control method of claim 30, wherein the neural network is a neural network trained in a sparsity promoting manner which uses as few neurons as possible based on another input data.