# US 2024/0256867 A1 · Neural network system that executes batch normalization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0256867 A1
- Document ID: US20240256867A1
- Status: Published application
- Kind: A1
- Application number: 18/560,798
- Filed: 2021-09-24
- Published: 2024-08-01
- Earliest claimed: 2021-06-09
- Assignees: Konica Minolta, Inc.
- Applicants: Konica Minolta, Inc.
- Inventors: Taiki Sekii
- Web page: https://patentgrep.com/patent/US20240256867A1

## CPC

- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/08: Learning methods
- G06F 16/906: Clustering; Classification
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A neural network system includes: a first layer generating first layer outputs of pieces of training data; a second layer; and a numerical conversion layer. During training, the numerical conversion layer: receives the first layer outputs from the first layer, calculates a numerical conversion parameter corresponding to each of the pieces of training data, numerically converts each of components of each of the first layer outputs using the numerical conversion parameter to generate a numerical conversion layer output, and inputs the same to the second layer. The numerical conversion parameter corresponding to one of the pieces of training data is: calculated from the first layer outputs of the pieces of training data except the one of the pieces of training data, or calculated by weighting one of the first layer outputs of the pieces of training data including the one of the pieces of training data.

## Claims

### Claim 1

A neural network system implemented by one or more computers, comprising:
a first layer that generates first layer outputs of a plurality of pieces of training data,
each of the first layer outputs having a plurality of components;
a second layer; and
a numerical conversion layer disposed between the first layer and the second layer, wherein
during training of the neural network system, the numerical conversion layer:
receives the first layer outputs from the first layer,
calculates a numerical conversion parameter corresponding to each of the pieces of training data,
numerically converts each of the components of each of the first layer outputs using the numerical conversion parameter to generate a numerical conversion layer output, and
inputs the numerical conversion layer output to the second layer, and
the numerical conversion parameter corresponding to one of the pieces of training data is:
calculated from the first layer outputs of the pieces of training data except the one of the pieces of training data, or
calculated by weighting each of the first layer outputs of the pieces of training data including the one of the pieces of training data, and
a weight of one of the first layer outputs of the one of the pieces of training data is smaller than a weight of the other first layer outputs of the other pieces of training data.

### Claim 2 (depends on 1)

The neural network system according to claim 1, wherein the numerical conversion parameter corresponding to the one of the pieces of training data is calculated from first layer outputs of a plurality of pieces of training data selected, from a batch including a set of the pieces of training data including the one of the pieces of training data, by a predetermined selection method to exclude the one of the pieces of training data.

### Claim 3 (depends on 2)

The neural network system according to claim 2, wherein
the components of each of the first layer outputs are indexed by dimensions, and
the numerical conversion layer calculates the numerical conversion parameter by:
with respect to the pieces of training data per the batch, for each of the dimensions, calculating a mean of the components of each of the first layer outputs of the pieces of training data selected by the selection method, as a pseudo mean of the components of each of the first layer outputs, and
calculating, for each of the dimensions, a variance of the components of each of the first layer outputs using the components of each of the first layer outputs and the pseudo mean with respect to the pieces of training data per the batch.

### Claim 4 (depends on 3)

The neural network system according to claim 3, wherein the numerical conversion layer generates the numerical conversion layer output by, for each of the pieces of training data, numerically converting the components of each of the first layer outputs of the pieces of training data using the pseudo mean and the variance for each of the dimensions corresponding to each of the components.

### Claim 5 (depends on 4)

The neural network system according to claim 4, wherein the numerical conversion layer generates the numerical conversion layer output by of transforming the components numerically converted based on values of a set of transformation parameters for each of the dimensions.

### Claim 6 (depends on 5)

The neural network system according to claim 5, wherein
after the neural network system is trained, the numerical conversion layer:
receives a new first layer output for a new neural network input generated by the first layer,
generates a new numerically converted layer output by numerically converting each of components of the new first layer output using a precalculated numerical conversion parameter,
generates a new numerical conversion layer output by converting, for each of the dimensions, each of components of the new numerically converted layer output based on a set of transformation parameters for each of the dimensions, and
newly inputs the new numerical conversion layer output to the second layer.

### Claim 7 (depends on 6)

The neural network system according to claim 6, wherein the precalculated numerical conversion parameter is calculated from the first layer outputs generated by the first layer during the training of the neural network system.

### Claim 8 (depends on 6)

The neural network system according to claim 6, wherein the precalculated numerical conversion parameter is calculated from the new first layer output generated by the first layer after the neural network system is trained.

### Claim 9 (depends on 7)

The neural network system according to claim 7, wherein a new neural network input processed by the neural network system after the neural network system is trained is an input of a different type from the pieces of training data used to train the neural network system.

### Claim 10 (depends on 2)

The neural network system according to claim 2, wherein
the components of each of the first layer outputs are indexed by a feature index and a spatial location index,
the numerical conversion layer calculates the numerical conversion parameter by:
with respect to the pieces of training data per the batch, for each combination of the feature index and the spatial location index, calculating a mean of the components of each of the first layer outputs of the pieces of training data selected by the selection method,
with respect to the pieces of training data per the batch, for each of the feature index, calculating an arithmetic mean of the mean with respect to the combination including the feature index,
for each combination of the feature index and the spatial location index, calculating a variance of the components of each of the first layer outputs using the components of each of the first layer outputs and the arithmetic mean with respect to the pieces of training data per the batch, and
for each of the feature index, calculating an arithmetic variance of the variance with respect to the combination including the feature index.

### Claim 11 (depends on 10)

The neural network system according to claim 10, wherein the numerical conversion layer generates the numerical conversion layer output by, for each of the pieces of training data, numerically converting the components of each of the first layer outputs of the pieces of training data using the arithmetic mean and the arithmetic variance.

### Claim 12 (depends on 11)

The neural network system according to claim 11, wherein the numerical conversion layer generates the numerical conversion layer output by converting the components numerically converted based on a set of transformation parameters for each of the feature index.

### Claim 13 (depends on 12)

The neural network system according to claim 12, wherein
after the neural network system is trained, the numerical conversion layer:
receives a new first layer output for a new neural network input generated by the first layer,
generates a new numerically converted layer output by numerically converting each of components of the new first layer output using a precalculated numerical conversion parameter,
generates a new numerical conversion layer output by converting, for each of the feature index, each of components of the new numerically converted layer output based on a set of transformation parameters for each of the feature index, and
newly inputs the new numerical conversion layer output to the second layer.

### Claim 14 (depends on 2)

The neural network system according to claim 2, wherein
the components of each of the first layer outputs are indexed by a feature index and a spatial location index, and
the numerical conversion layer calculates the numerical conversion parameter by:
with respect to the pieces of training data per the batch, for each of the feature index, calculating a mean of the components of each of the first layer outputs of the pieces of training data selected by the selection method, as a pseudo mean of the components of each of the first layer outputs, and
for each of the feature index, calculating a variance of the components of each of the first layer outputs using the components of each of the first layer outputs and the pseudo mean with respect to the pieces of training data per the batch.

### Claim 15 (depends on 14)

The neural network system according to claim 14, wherein the numerical conversion layer generates the numerical conversion layer output by, for each of the pieces of training data, numerically converting the components of each of the first layer outputs of the pieces of training data using the pseudo mean and the variance for the feature index corresponding to each of the components.

### Claim 16 (depends on 15)

The neural network system according to claim 15, wherein the numerical conversion layer generates the numerical conversion layer output by transforming the components numerically converted based on values of a set of transformation parameters for each of the feature index.

### Claim 17 (depends on 16)

The neural network system according to claim 16, wherein
after the neural network system is trained, the numerical conversion layer;
receives a new first layer output for a new neural network input generated by the first layer,
generates a new numerically converted layer output by numerically converting each of components of the new first layer output using a precalculated numerical conversion parameter,
generates a new numerical conversion layer output by converting, for each of the feature index, each of components of the new numerically converted layer output based on a set of transformation parameters for each of the feature index, and
newly inputs the new numerical conversion layer output to the second layer.

### Claim 18 (depends on 1)

The neural network system according to claim 1, wherein the first layer generates each of the first layer outputs by modifying a first layer input based on a set of parameters for the first layer.

### Claim 19 (depends on 18)

The neural network system according to claim 18, wherein the second layer generates a second layer output by applying a non-linear operation to a batch normalization layer output.

### Claim 20 (depends on 1)

The neural network system according to claim 1, wherein the first layer is a first neural network layer that generates each of the first layer outputs by modifying a first layer input based on current values of a set of parameters to generate a modified first layer input, and then applying a non-linear operation to the modified first layer input.

### Claim 21 (depends on 1)

The neural network system according to claim 1, wherein during the training of the neural network system, the neural network system back propagates the numerical conversion parameter for partially adjusting a value of a parameter of the neural network system.

### Claim 22 (depends on 2)

The neural network system according to claim 2, wherein the predetermined selection method selects, from the batch, some pieces of training data except the one of the pieces of training data, or all the pieces of training data.

### Claim 23 (depends on 1)

A method comprising:
executing an operation implemented by the numerical conversion layer according to claim 1.

### Claim 24 (depends on 1)

A non-transitory computer readable storage medium storing instructions executed by one or more computers, the instructions causing the one or more computers to function as the neural network system according to claim 1.
