# US 2023/0274132 A1 · Methods and apparatus to dynamically normalize data in neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274132 A1
- Document ID: US20230274132A1
- Status: Published application
- Kind: A1
- Application number: 18/005,804
- Filed: 2020-08-26
- Published: 2023-08-31
- Earliest claimed: 2020-08-26
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Dongqi Cai; Anbang Yao; Yurong Chen
- Web page: https://patentgrep.com/patent/US20230274132A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, apparatus, systems, and articles of manufacture to dynamically normalize data in neural networks are disclosed. An apparatus for use with a machine learning model includes at least one normalization calculator to generate a plurality of alternate normalized outputs associated with input data for the machine learning model. Different ones of the alternate normalized outputs based on different normalization techniques. The apparatus further includes a soft weighting engine to generate a plurality of soft weights based on the input data. The apparatus also includes a normalized output generator to generate a final normalized output based on the plurality of alternate normalized outputs and the plurality of soft weights.

## Claims

### Claim 1

An apparatus for use with a machine learning model, the apparatus comprising:
at least one normalization calculator to generate a plurality of alternate normalized outputs associated with input data for the machine learning model, different ones of the alternate normalized outputs based on different normalization techniques;
a soft weighting engine to generate a plurality of weights based on the input data; and
a normalized output generator to generate a final normalized output based on the plurality of alternate normalized outputs and the plurality of weights.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the normalized output generator is to generate the final normalized output as a sum of a product of ones of the plurality of weights and respective ones of the plurality of alternate normalized outputs.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the input data is first input data and the plurality of weights is a first plurality of weights, the soft weighting engine to generate a second plurality of weights based on second input data different than the first input data, the second plurality of weights different than the first plurality of weights due to distinctions between the first input data and the second input data.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the soft weighting engine includes:
an aggregation analyzer to aggregate the input data into a first vector; and
a mapping analyzer to map the first vector to a second vector, a number of elements in the second vector being the same as a number of the different normalization techniques, the plurality of weights based on values in the second vector.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the soft weighting engine includes a scaling analyzer to scale the values in the second vector.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the machine learning model is a neural network with multiple layers.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein the input data is first input data for a first layer in the neural network and the plurality of weights is a first plurality of weights, the soft weighting engine to generate a second plurality of weights based on second input data for a second layer in the neural network, the second input data based on the final normalized output.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein the plurality of alternate normalized outputs is a first plurality of alternate normalized outputs associated with the first layer in the neural network and the final normalized output is a first final normalized output associated with the first layer in the neural network, the at least one normalization calculator to generate a second plurality of alternate normalized outputs associated with second input data, the normalized output generator to generate a second final normalized output based on the second plurality of alternate normalized outputs and the second plurality of weights.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the soft weighting engine is to generate the plurality of weights independent of the alternate normalized outputs.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the plurality of weights corresponds to soft weights with values that may differ along a range from 0 to 1.

### Claim 11

At least one non-transitory computer readable medium comprising instructions that, when executed, cause at least one processor to at least:
generate a plurality of alternate normalized outputs associated with input data for a machine learning model, different ones of the alternate normalized outputs based on different normalization techniques;
generate a plurality of weights based on the input data; and
generate a final normalized output based on the plurality of alternate normalized outputs and the plurality of weights.

### Claim 12 (depends on 11)

The at least one non-transitory computer readable medium of claim 11, wherein the instructions further cause the at least one processor to generate the final normalized output as a sum of a product of ones of the plurality of weights and respective ones of the plurality of alternate normalized outputs.

### Claim 13 (depends on 11)

The at least one non-transitory computer readable medium of claim 11, wherein the input data is first input data and the plurality of weights is a first plurality of weights, the instructions to further cause the at least one processor to generate a second plurality of weights based on second input data different than the first input data, the second plurality of weights different than the first plurality of weights due to distinctions between the first input data and the second input data.

### Claim 14 (depends on 11)

The at least one non-transitory computer readable medium of claim 11, wherein the instructions further cause the at least one processor to:
aggregate the input data into a first vector; and
map the first vector to a second vector, a number of elements in the second vector being the same as a number of the different normalization techniques, the plurality of weights based on values in the second vector.

### Claim 15 (depends on 14)

The at least one non-transitory computer readable medium of claim 14, wherein the instructions further cause the at least one processor to scale the values in the second vector.

### Claim 16 (depends on 11)

The at least one non-transitory computer readable medium of claim 11, wherein the machine learning model is a neural network with multiple layers.

### Claim 17 (depends on 16)

The at least one non-transitory computer readable medium of claim 16, wherein the input data is first input data for a first layer in the neural network and the plurality of weights is a first plurality of weights, the instructions to further cause the at least one processor to generate a second plurality of weights based on second input data for a second layer in the neural network, the second input data based on the final normalized output.

### Claim 18 (depends on 17)

The at least one non-transitory computer readable medium of claim 17, wherein the plurality of alternate normalized outputs is a first plurality of alternate normalized outputs associated with the first layer in the neural network and the final normalized output is a first final normalized output associated with the first layer in the neural network, the instructions to further cause the at least one processor to:
generate a second plurality of alternate normalized outputs associated with second input data; and
generate a second final normalized output based on the second plurality of alternate normalized outputs and the second plurality of weights.

### Claim 19

A method for using a machine learning model, the method comprising:
generating a plurality of alternate normalized outputs associated with input data for the machine learning model, different ones of the alternate normalized outputs based on different normalization techniques;
generating a plurality of weights based on the input data; and
generating a final normalized output based on the plurality of alternate normalized outputs and the plurality of weights.

### Claim 20 (depends on 19)

The method of claim 19, further including generating the final normalized output as a sum of a product of ones of the plurality of weights and respective ones of the plurality of alternate normalized outputs.

### Claim 21 (depends on 19)

The method of claim 19, wherein the input data is first input data and the plurality of weights is a first plurality of weights, the method further including generating a second plurality of weights based on second input data different than the first input data, the second plurality of weights different than the first plurality of weights due to distinctions between the first input data and the second input data.

### Claim 22 (depends on 19)

The method of claim 19, further including:
aggregating the input data into a first vector; and
mapping the first vector to a second vector, a number of elements in the second vector being the same as a number of the different normalization techniques, the plurality of weights based on values in the second vector.

### Claim 23

An apparatus for use with a machine learning model, the apparatus comprising:
means for generating a plurality of alternate normalized outputs associated with input data for the machine learning model, different ones of the alternate normalized outputs based on different normalization techniques;
means for generating a plurality of weights based on the input data; and
means for generating a final normalized output based on the plurality of alternate normalized outputs and the plurality of weights.

### Claim 24 (depends on 23)

The apparatus of claim 23, wherein the final normalized output generating means is to generate the final normalized output as a sum of a product of ones of the plurality of weights and respective ones of the plurality of alternate normalized outputs.

### Claim 25 (depends on 23)

The apparatus of claim 23, wherein the input data is first input data and the plurality of weights is a first plurality of weights, the weights generating means to generate a second plurality of weights based on second input data different than the first input data, the second plurality of weights different than the first plurality of weights due to distinctions between the first input data and the second input data.
