# US 11,443,438 B2 · Network module and distribution method and apparatus, electronic device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,438 B2
- Document ID: US11443438B2
- Status: Granted
- Kind: B2
- Application number: 16/888,931
- Filed: 2020-06-01
- Published: 2020-09-17
- Granted: 2022-09-13
- Earliest claimed: 2019-02-25
- Assignees: Shenzhen Sensetime Technology Co., Ltd.
- Applicants: Shenzhen Sensetime Technology Co., Ltd.
- Inventors: Yi Li; Zhanghui Kuang; Yimin Chen; Wei Zhang
- Web page: https://patentgrep.com/patent/US11443438B2

## CPC

- G06N 3/063: using electronic means
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06T 7/35: using statistical methods
- G06T 2207/20084: Artificial neural networks [ANN]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A distribution method includes: distributing, when multiple feature maps exist in an image processing model, to each feature map a neuron through which the feature map passes; filtrating, according to importance of neurons of multiple convolution layers in an image processing model, the neurons to obtain a first result; collecting, according to a position attribute of the each neuron in the first result, statistics on a scale of the feature map corresponding to each neuron to obtain a distribution relationship; the distribution relationship indicates a correspondence between the each feature map and the neuron through which the feature map passes; and distributing, according to the distribution relationship, to the each feature map the neuron through which the feature map passes.

## Claims

### Claim 1

A distribution method, comprising:
distributing, when multiple feature maps exist in an image processing model, to each feature map a neuron through which the feature map passes;
filtrating, according to importance of neurons of multiple convolution layers in an image processing model, the neurons to obtain a first result;
wherein the multiple convolution layers are located at a same depth of the image processing model, and
the multiple convolution layers are respectively configured to process the feature maps of different scales;
the first result comprises multiple neurons;
collecting, according to a position attribute of the each neuron in the first result, statistics on a scale of the feature map corresponding to each neuron to obtain a distribution relationship;
wherein the position attribute indicates a convolution layer to which the each neuron belongs;
the distribution relationship indicates a correspondence between the each feature map and the neuron through which the feature map passes; and
distributing, according to the distribution relationship, to the each feature map the neuron through which the feature map passes.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the image processing model comprises at least one network module, and each network module comprises:
a first network layer, a second network layer, and a third network layer that are cascaded sequentially;
wherein the first network layer is configured to process an inputted image to obtain a first feature map;
the second network layer comprises multiple parallel branches;
each branch of the multiple parallel branches comprises a first sampling layer;
the first sampling layer is configured to perform downsampling on the first feature map to obtain a second feature map;
wherein scales of second feature maps obtained by first sampling layers in different branches of the multiple parallel branches are different;
the third network layer is configured to merge feature map data outputted by the each branch.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the first sampling layer is a pooling layer.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the pooling layer is a maximum pooling layer.

### Claim 5 (depends on 2)

The method according to claim 2, wherein the each branch further comprises a first convolution layer and a second sampling layer that are cascaded sequentially with the first sampling layer;
the first convolution layer is configured to perform a convolution operation on the second feature map to obtain a third feature map;
the second sampling layer is configured to restore a scale of the third feature map to a scale of the first feature map.

### Claim 6 (depends on 2)

The method according to claim 2, wherein the second network layer further comprises an original proportional branch; the original proportional branch comprises a second convolution layer;
the second convolution layer is configured to perform a convolution operation on the first feature map, and input feature map data obtained by the convolution operation into the third network layer;
the third network layer is further configured to merge feature map data outputted by the multiple parallel branches in the second network layer with feature map data outputted by the original proportional branch.

### Claim 7 (depends on 1)

The method according to claim 1, before filtrating, according to the importance of the neurons of the multiple convolution layers in the image processing model, the neurons, further comprising:
determining a number of branches in each network module, and constructing the image processing model according to a preset number of network modules; and
training the image processing model to obtain a scale parameter of a batchnorm layer in the image processing model;
wherein the scale parameter indicates the importance of the neurons of the convolution layers in the branches in the network module.

### Claim 8 (depends on 1)

The method according to claim 1, wherein filtrating, according to the importance of the neurons of the multiple convolution layers in the image processing model, the neurons to obtain the first result comprises:
sorting, according to a scale parameter of a batchnorm layer obtained by pre-training the image processing model, the neurons of the multiple convolution layers to obtain a first sequence;
wherein the first sequence indicates an arrangement order of the neurons of the multiple convolution layers;
determining a number of neurons to be used for processing the multiple feature maps according to a preset amount of calculation; and
sequentially extracting, according to a determined number of the neurons to be used, required neurons from the first sequence to obtain the first result.

### Claim 9 (depends on 8)

The method according to claim 8, after distributing, according to the distribution relationship, to the each feature map the neuron through which the feature map passes, further comprising:
determining, according to the distribution relationship, a first network structure of the image processing model; and
training the image processing model of the first network structure.

### Claim 10

An electronic device, comprising:
a processor; and
a memory configured to store processor-executable instructions;
wherein when the processor-executable instructions are executed, the processor is configured to:
distribute, when multiple feature maps exist in an image processing model, to each feature map a neuron through which the feature map passes;
filtrate neurons according to importance of the neurons of multiple convolution layers in the image processing model to obtain a first result;
wherein the multiple convolution layers are located at a same depth of the image processing model, and
the multiple convolution layers are respectively configured to process the feature maps of different scales;
the first result comprises multiple neurons;
collect statistics on a scale of the feature map corresponding to each neuron according to a position attribute of the each neuron in the first result to obtain a distribution relationship;
wherein the position attribute indicates a convolution layer to which the each neuron belongs;
the distribution relationship indicates a correspondence between the each feature map and the neuron through which the feature map passes; and
distribute, according to the distribution relationship, to the each feature map the neuron through which the feature map passes.

### Claim 11 (depends on 10)

The electronic device according to claim 10, wherein the image processing model comprises at least one network module, and each network module comprises:
a first network layer, a second network layer, and a third network layer that are cascaded sequentially;
wherein the first network layer is configured to process an inputted image to obtain a first feature map;
the second network layer comprises multiple parallel branches;
each branch of the multiple parallel branches comprises a first sampling layer;
the first sampling layer is configured to perform downsampling on the first feature map to obtain a second feature map;
wherein scales of second feature maps obtained by first sampling layers in different branches of the multiple parallel branches are different;
the third network layer is configured to merge feature map data outputted by the each branch.

### Claim 12 (depends on 11)

The electronic device according to claim 11, wherein the first sampling layer is a pooling layer.

### Claim 13 (depends on 12)

The electronic device according to claim 12, wherein the pooling layer is a maximum pooling layer.

### Claim 14 (depends on 11)

The electronic device according to claim 11, wherein the each branch further comprises a first convolution layer and a second sampling layer that are cascaded sequentially with the first sampling layer;
the first convolution layer is configured to perform a convolution operation on the second feature map to obtain a third feature map;
the second sampling layer is configured to restore a scale of the third feature map to a scale of the first feature map.

### Claim 15 (depends on 11)

The electronic device according to claim 11, wherein the second network layer further comprises an original proportional branch; the original proportional branch comprises a second convolution layer;
the second convolution layer is configured to perform a convolution operation on the first feature map, and input feature map data obtained by the convolution operation into the third network layer;
the third network layer is further configured to merge feature map data outputted by the multiple parallel branches in the second network layer with feature map data outputted by the original proportional branch.

### Claim 16 (depends on 10)

The electronic device according to claim 10, wherein the processor is further configured to:
determine a number of branches in each network module, and construct the image processing model according to a preset number of network modules before the processor filtrates the neurons according to the importance of the neurons of the multiple convolution layers in the image processing model;
train the image processing model to obtain a scale parameter of a batchnorm layer in the image processing model;
wherein the scale parameter indicates the importance of the neurons of the convolution layers in the branches in the network module.

### Claim 17 (depends on 10)

The electronic device according to claim 10, wherein the processor is specifically configured to:
sort the neurons of the multiple convolution layers according to a scale parameter of a batchnorm layer obtained by pre-training the image processing model to obtain a first sequence;
wherein the first sequence indicates an arrangement order of the neurons of the multiple convolution layers;
determine a number of neurons to be used for processing the multiple feature maps according to a preset amount of calculation; and
sequentially extract required neurons from the first sequence according to a determined number of the neurons to be used to obtain the first result.

### Claim 18 (depends on 17)

The electronic device according to claim 17, wherein the processor is further configured to:
determine a first network structure of the image processing model according to the distribution relationship after the processor distributes, according to the distribution relationship, to the each feature map the neuron through which the feature map passes; and
train the image processing model of the first network structure.

### Claim 19

A non-transitory computer-readable storage medium, having computer program instructions thereon, wherein when the computer program instructions are executed by a processor, the processor is configured to perform:
distributing, when multiple feature maps exist in an image processing model, to each feature map a neuron through which the feature map passes;
filtrating, according to importance of neurons of multiple convolution layers in an image processing model, the neurons to obtain a first result;
wherein the multiple convolution layers are located at a same depth of the image processing model, and
the multiple convolution layers are respectively configured to process the feature maps of different scales;
the first result comprises multiple neurons;
collecting, according to a position attribute of the each neuron in the first result, statistics on a scale of the feature map corresponding to each neuron to obtain a distribution relationship;
wherein the position attribute indicates a convolution layer to which the each neuron belongs;
the distribution relationship indicates a correspondence between the each feature map and the neuron through which the feature map passes; and
distributing, according to the distribution relationship, to the each feature map the neuron through which the feature map passes.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium according to claim 19, wherein the image processing model comprises at least one network module, and each network module comprises:
a first network layer, a second network layer, and a third network layer that are cascaded sequentially;
wherein the first network layer is configured to process an inputted image to obtain a first feature map;
the second network layer comprises multiple parallel branches;
each branch of the multiple parallel branches comprises a first sampling layer;
the first sampling layer is configured to perform downsampling on the first feature map to obtain a second feature map;
wherein scales of second feature maps obtained by first sampling layers in different branches of the multiple parallel branches are different;
the third network layer is configured to merge feature map data outputted by the each branch.