US 12,755,131 B2Granted

Normalization in deep convolutional neural networks

AssigneeHuawei Technologies Co., Ltd.

InventorsXiaoyun Zhou, Jiacheng Sun, Nanyang Ye, Xu Lan, Qijun Luo, Pedro Esperanca, Fabio Maria Carlucci, Zewei Chen, Zhenguo Li

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

A device for machine learning is provided, including a first neural network layer, a second neural network layer with a normalization layer arranged in between. The normalization layer is configured to, when the device is undergoing training on a batch of training samples, receive multiple outputs of the first neural network layer for a plurality of training samples of the batch, each output comprising multiple data values for different indices on a first dimension and a second dimension; group the outputs into multiple groups based on the indices on the first and second dimensions; form a normalization output for each group which are provided as input to the second neural network layer. According to the application, the training of a deep convolutional neural network with good performance that performs stably at different batch sizes and is generalizable to multiple vision tasks is achieved, thereby improving the performance of the training.

Claims20

Dependent claims

A device for machine learning configured to implement a deep convolutional neural network (DCNN), the device comprising:

one or more processors;
a first neural network layer;
a second neural network layer;
and
a normalization layer arranged between the first neural network layer and the second neural network layer;
wherein when the device is undergoing training on a batch of training samples, the one or more processors cooperate with the normalization layer and cause the normalization layer to:
receive multiple outputs of the first neural network layer for a plurality of training samples of the batch of training samples,
wherein the multiple outputs comprise a plurality of feature maps produced by the first neural network layer,
wherein each output comprises multiple data values for indices of a feature map among the plurality of feature maps along a first dimension and a second dimension,
wherein the first dimension represents a channel dimension that indexes a plurality of channels, each channel corresponding to a respective feature map in the plurality of feature maps produced by the first neural network layer, and
wherein the second dimension represents at least one spatial dimension of each feature map in the plurality of feature maps, including a height dimension and/or a width dimension;
group the outputs into multiple groups of features based on at least the multiple data values for different indices on the first and second dimensions to which the outputs relate;
form a normalization output for each group of outputs;
and
provide the normalization outputs as input to the second neural network layer.

12 more elements

A method for training a device for machine learning configured to implement a deep convolutional neural network (DCNN), wherein the training occurs on a batch of training samples, wherein the DCNN comprises a first neural network layer, a second neural network layer and a normalization layer arranged between the first neural network layer and the second neural network layer, the method comprising the following operations carried out by the normalization layer:

receiving multiple outputs of the first neural network layer for a plurality of training samples of the batch of training samples,
wherein the multiple outputs comprise a plurality of feature maps produced by the first neural network layer,
wherein each output comprises multiple data values for indices of a feature map among the plurality of feature maps along a first dimension and a second dimension,
wherein the first dimension represents a channel dimension that indexes a plurality of channels, each channel corresponding to a respective feature map in the plurality of feature maps produced by the first neural network layer, and
wherein the second dimension represents at least one spatial dimension of each feature map in the plurality of feature maps, including a height dimension and/or a width dimension;
grouping the outputs into multiple groups of features based on at least the multiple data values for different indices on the first and second dimensions to which the outputs relate;
forming a normalization output for each group of outputs;
and
providing the normalization outputs as input to the second neural network layer.

6 more elements

Dependencies are inferred from the claim text.