# US 11,308,391 B2 · Offline combination of convolutional/deconvolutional and batch-norm layers of convolutional neural network models for autonomous driving vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,391 B2
- Document ID: US11308391B2
- Status: Granted
- Kind: B2
- Application number: 15/451,345
- Filed: 2017-03-06
- Published: 2018-09-06
- Granted: 2022-04-19
- Earliest claimed: 2017-03-06
- Assignees: Baidu USA LLC
- Applicants: Baidu USA LLC
- Inventors: Zhenhua Yu; Xiao Bo; Jun Zhou; Weide Zhang; Tony Han
- Web page: https://patentgrep.com/patent/US11308391B2

## CPC

- G06N 3/08: Learning methods
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/584: {of vehicle lights or traffic lights}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/048: Activation functions
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In one embodiment, a system to accelerate batch-normalized convolutional neural network (CNN) models is disclosed. The system extracts a plurality of first groups of layers from a first CNN model, each group of the first groups having a first convolutional layer and a first batch-norm layer. For each group of the plurality of first groups, the system calculates a first scale vector and a first shift vector based on the first batch-norm layer, and generates a second convolutional layer representing the corresponding group of the plurality of first groups based on the first convolutional layer and the first scale and the first shift vectors. The system generates an accelerated CNN model based on the second convolutional layer corresponding to the plurality of the first groups, such that the accelerated CNN model is utilized subsequently to classify an object perceived by an autonomous driving vehicle (ADV).

## Claims

### Claim 1

A computer-implemented method, the method comprising:
receiving, at a neural network model generator, a first convolutional neural network (CNN) model as input to the neural network model generator, wherein the first CNN model is a fully trained CNN model;
extracting, by the neural network model generator, a plurality of first groups of layers and a plurality of second groups of layers from the first convolutional neural network (CNN) model, each group of the plurality of first groups having a first convolutional layer, a first batch-norm layer, and a first scaling layer that is separate from the first batch-norm layer, and each group of the plurality of second groups having a first deconvolutional layer and a second batch-norm layer;
for each group of the plurality of first groups,
calculating, by the neural network model generator, a first scale vector and a first shift vector based on the first batch-norm layer, wherein the first scale vector and the first shift vector are calculated scalarly for each dimension of a multi-dimensional input of the first convolutional layer,
calculating a second scale vector and a second shift vector based on the first scaling layer,
combining the first scale vector and the second scale vector into a combined scale vector, and the first shift vector and the second shift vector into a combined shift vector, wherein each of the first scale vector and the first shift vector is calculated from the first batch-norm layer, and wherein each of the second scale vector and the second shift vector is calculated from the first scaling layer, and
generating, by the neural network model generator, a second convolutional layer representing the corresponding group of the plurality of first groups based on the first convolutional layer, the combined scale vector, and the combined shift vector;
for each group of the plurality of second groups,
calculating a third scale vector and a third shift vector based on the second batch-norm layer, and
generating a second deconvolutional layer representing the corresponding group of the plurality of second groups based on the third scale vector, the third shift vector, and the first deconvolutional layer; and
generating, by the neural network model generator, a second CNN model based on the second convolutional layer corresponding to each of the plurality of first groups and the deconvolutional layer corresponding to each of the plurality of second groups, wherein the second CNN model is utilized subsequently to classify an object perceived by an autonomous driving vehicle (ADV).

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
for each group of the plurality of second groups,
calculating a fourth scale vector and a fourth shift vector based on the second batch-norm layer and a second scaling layer of the corresponding group of the plurality of second groups; and
generating the second deconvolutional layer based on the fourth scale vector, the fourth shift vector, and the first deconvolutional layer.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the first scale vector and the first shift vector are calculated based on means and standard deviations of each dimension of the multi-dimensional input.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the mean and the standard deviation are calculated for each dimension of the multi-dimensional input independently.

### Claim 5 (depends on 1)

The method of claim 1, wherein the neural network model generator runs on a server that the ADV is communicatively coupled to.

### Claim 6

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
receiving, at a neural network model generator, a first convolutional neural network (CNN) model as input to the neural network model generator, wherein the first CNN model is a fully trained CNN model;
extracting, by the neural network model generator, a plurality of first groups of layers and a plurality of second groups of layers from the first convolutional neural network (CNN) model, each group of the plurality of first groups having a first convolutional layer, a first batch-norm layer, and a first scaling layer that is separate from the first batch-norm layer, and each group of the plurality of second groups having a first deconvolutional layer and a second batch-norm layer;
for each group of the plurality of first groups,
calculating, by the neural network model generator, a first scale vector and a first shift vector based on the first batch-norm layer, wherein the first scale vector and the first shift vector are calculated scalarly for each dimension of a multi-dimensional input of the first convolutional layer,
calculating a second scale vector and a second shift vector based on the first scaling layer,
combining the first scale vector and the second scale vector into a combined scale vector, and the first shift vector and the second shift vector into a combined shift vector, wherein each of the first scale vector and the first shift vector is calculated from the first batch-norm layer, and wherein each of the second scale vector and the second shift vector is calculated from the first scaling layer, and
generating, by the neural network model generator, a second convolutional layer representing the corresponding group of the plurality of first groups based on the first convolutional layer, the combined scale vector, and the combined shift vector;
for each group of the plurality of second groups,
calculating a third scale vector and a third shift vector based on the second batch-norm layer, and
generating a second deconvolutional layer representing the corresponding group of the plurality of second groups based on the third scale vector, the third shift vector, and the first deconvolutional layer; and
generating, by the neural network model generator, a second CNN model based on the second convolutional layer corresponding to each of the plurality of first groups and the deconvolutional layer corresponding to each of the plurality of second groups, wherein the second CNN model is utilized subsequently to classify an object perceived by an autonomous driving vehicle (ADV).

### Claim 7 (depends on 6)

The non-transitory machine-readable medium of claim 6, further comprising:
for each group of the plurality of second groups,
calculating a fourth scale vector and a fourth shift vector based on a second scaling layer of the corresponding group of the plurality of second groups; and
generating the second deconvolutional layer based on the fourth scale vector, the fourth shift vector, and the first deconvolutional layer.

### Claim 8 (depends on 6)

The non-transitory machine-readable medium of claim 6, the first scale vector and the first shift vector are calculated based on means and standard deviations of each dimension of the multi-dimensional input.

### Claim 9 (depends on 8)

The non-transitory machine-readable medium of claim 8, wherein the mean and the standard deviation are calculated for each dimension of the multi-dimensional input independently.

### Claim 10 (depends on 6)

The non-transitory machine-readable medium of claim 6, wherein the neural network model generator runs on a server that the ADV is communicatively coupled to.

### Claim 11

A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including:
receiving, at a neural network model generator, a first convolutional neural network (CNN) model as input to the neural network model generator, wherein the first CNN model is a fully trained CNN model,
extracting a plurality of first groups of layers and a plurality of second groups of layers from the first convolutional neural network (CNN) model, each group of the plurality of first groups having a first convolutional layer, a first batch-norm layer, and a first scaling layer that is separate from the first batch-norm layer, and each group of the plurality of second groups having a first deconvolutional layer and a second batch-norm layer,
for each group of the plurality of first groups,
calculating a first scale vector and a first shift vector based on the first batch-norm layer, wherein the first scale vector and the first shift vector are calculated scalarly for each dimension of a multi-dimensional input of the first convolutional layer,
calculating a second scale vector and a second shift vector based on the first scaling layer,
combining the first scale vector and the second scale vector into a combined scale vector, and the first shift vector and the second shift vector into a combined shift vector, wherein each of the first scale vector and the first shift vector is calculated from the first batch-norm layer, and wherein each of the second scale vector and the second shift vector is calculated from the first scaling layer, and
generating a second convolutional layer representing the corresponding group of the plurality of first groups based on the first convolutional layer, the combined scale vector, and the combined shift vector,
for each group of the plurality of second groups,
calculating a third scale vector and a third shift vector based on the second batch-norm layer, and
generating a second deconvolutional layer representing the corresponding group of the plurality of second groups based on the third scale vector, the third shift vector, and the first deconvolutional layer,
generating a second CNN model based on the second convolutional layer corresponding to each of the plurality of first groups and the deconvolutional layer corresponding to each of the plurality of second groups, and
classifying an object perceived by an autonomous driving vehicle (ADV) using the second CNN model.

### Claim 12 (depends on 11)

The system of claim 11, further comprising:
for each group of the plurality of second groups,
calculating a fourth scale vector and a fourth shift vector based on the second batch-norm layer and a second scaling layer of the corresponding group of the plurality of second groups; and
generating the second deconvolutional layer based on the fourth scale vector, the fourth shift vector, and the first deconvolutional layer.

### Claim 13 (depends on 11)

The system of claim 11, wherein the first scale vector and the first shift vector are calculated based on means and standard deviations of each dimension of the multi-dimensional input.

### Claim 14 (depends on 13)

The system of claim 13, wherein the mean and the standard deviation are calculated for each dimension of the multi-dimensional input independently.

### Claim 15 (depends on 11)

The system of claim 11, wherein the neural network model generator runs on a server that the ADV is communicatively coupled to.
