# US 11,308,397 B2 · System and method of training a neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,397 B2
- Document ID: US11308397B2
- Status: Granted
- Kind: B2
- Application number: 16/274,319
- Filed: 2019-02-13
- Published: 2019-08-22
- Granted: 2022-04-19
- Earliest claimed: 2018-02-16
- Applicants: Ilya Sorokin
- Inventors: Ilya Sorokin
- Web page: https://patentgrep.com/patent/US11308397B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/048: Activation functions
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/0499: Feedforward networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/065: Analogue means
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A modified model of the Rosenblatt perceptron and learning method leading to a significant increase in the intelligence and learning speed of the Rosenblatt perceptron itself, as well as arbitrary neural network topologies based on it, based on the modified error correction method and the modified “Back propagation” algorithm. To increase the manageability of the perceptron topology, as well as the processes of recognition and learning, the cost of connection is added at the synapses in addition to the weighting factor. To increase the speed of the perceptron learning, first, at least at the initial stage of learning, the process is moved from nested recursive iterations of learning on different layers of synapses of perceptron to successive learning for different layers; and, second, the process is moved from the constant rate of learning parameter, which requires a sequence of small iterations when adjusting the weights, to the functional parameter of the learning speed, correcting the weight so that the error for the current image is compensated completely in one step.

## Claims

### Claim 1

A method of training a neural network having at least one neuron, the method comprising:
assigning a weight and a cost of connection to each signal inputted into said at least one neuron, wherein said cost of connection is any value between two predefined values; and
training said at least one neuron in a single step using a variable learning rate parameter to correct weights of said signals, said variable learning rate parameter being dependent on said assigned cost of connection, such that an error for a particular neuron is distributed among all synapses of said particular neuron.

### Claim 2 (depends on 1)

The method of claim 1, wherein said assigned cost of connection comprises a group of costs of connection.

### Claim 3 (depends on 1)

The method of claim 1, wherein said cost of connection is any value between 0 and 1.

### Claim 4 (depends on 1)

The method of claim 1, further comprising self-organizing said neural network by performing training of all layers of synapses except synapses with R-units as unsupervised learning.

### Claim 5 (depends on 4)

The method of claim 4, wherein said self-organizing is performed with extraction of essential features of all input images of a particular layer.

### Claim 6 (depends on 5)

The method of claim 5, wherein during said self-organizing a complete cycle of training of a first layer is carried out right up to stabilization of weights.

### Claim 7 (depends on 6)

The method of claim 6, wherein output images of a particular layer are input images for a subsequent layer, unless said particular layer is a layer of R-units.

### Claim 8 (depends on 1)

The method of claim 1, further comprising performing a final correction using a method suitable for error corrections.

### Claim 9 (depends on 8)

The method of claim 8, wherein said final correction is performed using a back propagation method.

### Claim 10

An apparatus for training a neural network having at least one neuron, the apparatus comprising:
a memory; and
at least one processor coupled to the memory, the at least one processor being configured to assign a weight and a cost of connection to each signal inputted into said at least one neuron, wherein said cost of connection is any value between two predefined values; and to train said at least one neuron in a single step using a variable learning rate parameter to correct weights of said signals, said variable learning rate parameter being dependent on said assigned cost of connection, such that an error for a particular neuron is distributed among all synapses of said particular neuron.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein said assigned cost of connection comprises a group of costs of connection.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein said cost of connection is any value between 0 and 1.

### Claim 13 (depends on 10)

The apparatus of claim 10, wherein said at least one processor is further configured to self-organize said neural network by performing training of all layers of synapses except synapses with R-units as unsupervised learning.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein said at least one processor is further configured to perform said self-organizing with extraction of essential features of all input images of a particular layer.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein said at least one processor is further configured to carry out a complete cycle of training of a first layer right up to stabilization of weights.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein during said self-organization, output images of said particular layer are input images for a subsequent layer, unless said particular layer is a layer of R-units.

### Claim 17 (depends on 10)

The apparatus of claim 10, wherein said at least one processor is further configured to perform a final correction using a method suitable for error corrections.

### Claim 18 (depends on 10)

The apparatus of claim 10, wherein said at least one processor is further configured to perform said final correction using a back propagation method.

### Claim 19

A non-transitory computer-readable medium having encoded thereon program code for training a neural network having at least one neuron, the program code being executed by a processor and comprising:
a first program code to assign a weight and a cost of connection to each signal inputted into said at least one neuron, wherein said cost of connection is any value between two predefined values; and
a second program code to train said at least one neuron in a single step using a variable learning rate parameter to correct weights of said signals, said variable learning rate parameter being dependent on said assigned cost of connection, such that an error for a particular neuron is distributed among all synapses of said particular neuron.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein said assigned cost of connection comprises a group of costs of connection.

### Claim 21 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein said cost of connection is any value between 0 and 1.

### Claim 22 (depends on 19)

The non-transitory computer-readable medium of claim 19, further comprising a third program code to self-organize said neural network by performing training of all layers of synapses except synapses with R-units as unsupervised learning.

### Claim 23 (depends on 22)

The non-transitory computer-readable medium of claim 22, wherein said third program code further comprises a program code to perform said self-organization with extraction of essential features of all input images of a particular layer.

### Claim 24 (depends on 23)

The non-transitory computer-readable medium of claim 23, wherein said third program code is configured to carry out a complete cycle of training of a first layer right up to stabilization of weights.

### Claim 25 (depends on 24)

The non-transitory computer-readable medium of claim 24, wherein said third program code is further configured to enter output images of said particular layer as input images for a subsequent layer, unless said particular layer is a layer of R-units.

### Claim 26 (depends on 19)

The non-transitory computer-readable medium of claim 19, further comprising a fourth program code configured to perform a final correction using a method suitable for error corrections.

### Claim 27 (depends on 26)

The non-transitory computer-readable medium of claim 26, wherein said fourth program code is configured to perform said final correction using a back propagation method.