# US 2022/0044370 A1 · Image processing methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044370 A1
- Document ID: US20220044370A1
- Status: Published application
- Kind: A1
- Application number: 17/388,043
- Filed: 2021-07-29
- Published: 2022-02-10
- Earliest claimed: 2020-08-05
- Assignees: Ubtech Robotics Corp Ltd
- Applicants: Ubtech Robotics Corp Ltd
- Inventors: Bin Sun; Mingguo Zhao; Youjun Xiong
- Web page: https://patentgrep.com/patent/US20220044370A1

## CPC

- G06T 5/20: using local operators
- G06V 10/82: using neural networks
- G06F 7/24: Sorting, i.e. extracting data from one or more carriers, rearranging the data in numerical or other ordered sequence, and rerecording the sorted data on the original carrier or on a different carrier or set of carriers {sorting methods in general}(G06F7/36 takes precedence)
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Image processing methods are provided. One of the method includes: obtaining a to-be-processed multi-channel feature maps; obtaining multi-channel first output feature maps and multi-channel second output feature maps by processing the multi-channel feature maps through a parallel pointwise convolution and non-pointwise operation, where the non-pointwise convolution is for descripting a spatial feature of each channel and an information exchange between the feature maps; and fusing the multi-channel first output feature maps and the multi-channel second output feature maps to obtain a multi-channel third output feature map.

## Claims

### Claim 1

An image processing method, comprising steps of:
obtaining to-be-processed multi-channel feature maps, wherein each of the multi-channel feature maps corresponding to a channel;
obtaining multi-channel first output feature maps by processing the multi-channel feature maps through a pointwise convolution and multi-channel second output feature maps by processing the multi-channel feature maps through a non-pointwise convolution, wherein the pointwise convolution and the non-pointwise convolution are performed in parallel, and the non-pointwise convolution is for descripting a spatial feature of each channel and an information exchange between the feature maps; and
fusing the multi-channel first output feature maps and the multi-channel second output feature maps to obtain multi-channel third output feature maps;
wherein, the non-pointwise convolution is an adaptive linear operation, and in response to a first ratio of a channel amount corresponding to the channel feature maps to perform the non-pointwise convolution and a channel amount corresponding to the multi-channel second output feature maps being a positive integer, the multi-channel second output feature maps is obtained by:
evenly dividing the channel feature maps into a plurality of feature map groups by channel according to the first ratio, wherein an amount of the feature maps in each feature map group is the first ratio; and
performing the linear operation on the feature map corresponding to each channel in each feature map group, and using a cumulative sum of results of the linear operations of the feature map group as the second output feature maps corresponding to the feature map group.

### Claim 2 (depends on 1)

The method of claim 1, wherein the obtaining the multi-channel first output feature maps by processing the multi-channel feature map through the pointwise convolution and the multi-channel second output feature maps by processing the multi-channel feature map through the non-pointwise convolution comprises:
obtaining the multi-channel first output feature maps by performing the pointwise convolution on the multi-channel feature maps; and
obtaining the multi-channel second output feature maps by performing the non-pointwise convolution on the multi-channel feature maps;
wherein the fusing the multi-channel first output feature maps and the multi-channel second output feature maps to obtain the multi-channel third output feature map comprises:
obtaining the multi-channel third output feature map by performing an accumulation operation on the multi-channel first output feature maps and the multi-channel second output feature maps.

### Claim 3 (depends on 1)

The method of claim 1, wherein the multi-channel feature maps are divided into a first part of the multi-channel feature maps and a second part of the multi-channel feature maps by channel, and the obtaining the multi-channel first output feature maps by processing the multi-channel feature map through the pointwise convolution and the multi-channel second output feature maps by processing the multi-channel feature map through the non-pointwise convolution comprises:
obtaining the multi-channel first output feature maps by performing the pointwise convolution on the first part of the multi-channel feature maps; and
obtaining the multi-channel second output feature maps by performing the non-pointwise convolution on the second part of the multi-channel feature maps;
wherein the fusing the multi-channel first output feature maps and the multi-channel second output feature maps to obtain the multi-channel third output feature map comprises:
obtaining the multi-channel third output feature map by performing a channel reorganization on the multi-channel first output feature maps and the multi-channel second output feature maps.

### Claim 4

An image processing method, comprising steps of:
obtaining to-be-processed multi-channel feature maps, wherein each of the multi-channel feature maps corresponding to a channel;
obtaining multi-channel first output feature maps by processing the multi-channel feature maps through a pointwise convolution and multi-channel second output feature maps by processing the multi-channel feature maps through a non-pointwise convolution, wherein the pointwise convolution and the non-pointwise convolution are performed in parallel, and the non-pointwise convolution is for descripting a spatial feature of each channel and an information exchange between the feature maps; and
fusing the multi-channel first output feature maps and the multi-channel second output feature maps to obtain a multi-channel third output feature map;
wherein, the non-pointwise convolution is an adaptive linear operation, and in response to a second ratio of a channel amount corresponding to the multi-channel second output feature maps to a channel amount corresponding to the channel feature maps to perform the non-pointwise convolution being a positive integer, the multi-channel second output feature maps is obtained by:
evenly dividing the channel feature maps into the feature map groups by channel, wherein an amount of the feature map in each feature map group is 1; and
performing the linear operation on the feature map in each feature map group for a plurality of times, and using results of the linear operation corresponding to the feature map as the second output feature maps of channels corresponding to the feature map, wherein the times of performing the linear operation on the feature map in each feature map group is equal to the second ratio.

### Claim 5 (depends on 4)

The method of claim 4, wherein the obtaining the multi-channel first output feature maps by processing the multi-channel feature map through the pointwise convolution and the multi-channel second output feature maps by processing the multi-channel feature map through the non-pointwise convolution comprises:
obtaining the multi-channel first output feature maps by performing the pointwise convolution on the multi-channel feature maps; and
obtaining the multi-channel second output feature maps by performing the non-pointwise convolution on the multi-channel feature maps;
wherein the fusing the multi-channel first output feature maps and the multi-channel second output feature maps to obtain the multi-channel third output feature map comprises:
obtaining the multi-channel third output feature map by performing an accumulation operation on the multi-channel first output feature maps and the multi-channel second output feature maps.

### Claim 6 (depends on 4)

The method of claim 4, wherein the multi-channel feature maps are divided into a first part of the multi-channel feature maps and a second part of the multi-channel feature maps by channel, and the obtaining the multi-channel first output feature maps by processing the multi-channel feature map through the pointwise convolution and the multi-channel second output feature maps by processing the multi-channel feature map through the non-pointwise convolution comprises:
obtaining the multi-channel first output feature maps by performing the pointwise convolution on the first part of the multi-channel feature maps; and
obtaining the multi-channel second output feature maps by performing the non-pointwise convolution on the second part of the multi-channel feature maps;
wherein the fusing the multi-channel first output feature maps and the multi-channel second output feature maps to obtain the multi-channel third output feature map comprises:
obtaining the multi-channel third output feature map by performing a channel reorganization on the multi-channel first output feature maps and the multi-channel second output feature maps.

### Claim 7

An image processing method, comprising steps of:
obtaining to-be-processed multi-channel feature maps, wherein each of the multi-channel feature maps corresponding to a channel;
obtaining multi-channel first output feature maps by processing the multi-channel feature maps through a pointwise convolution and multi-channel second output feature maps by processing the multi-channel feature maps through a non-pointwise convolution, wherein the pointwise convolution and the non-pointwise convolution are performed in parallel, and the non-pointwise convolution is for descripting a spatial feature of each channel and an information exchange between the feature maps; and
fusing the multi-channel first output feature maps and the multi-channel second output feature maps to obtain a multi-channel third output feature map;
wherein, the non-pointwise convolution is an adaptive linear operation, and in response to both a first ratio of a channel amount corresponding to the channel feature maps to perform the non-pointwise convolution and a channel amount corresponding to the multi-channel second output feature maps and a second ratio of a channel amount corresponding to the multi-channel second output feature maps to a channel amount corresponding to the channel feature maps to perform the non-pointwise convolution being not positive integers, the multi-channel second output feature maps is obtained by:
obtaining a common divisor between the channel amount of the channel feature maps and the channel amount of the multi-channel second output feature maps;
using one of the common divisors as a target common divisor;
grouping the channel feature maps into first feature map groups and the multi-channel second output feature maps into second output feature map groups by channel according to the target common divisor, wherein there are a plurality of feature maps in each first feature map group, and there are a plurality of second output feature maps in each second output feature map group, and an amount of the first feature map groups and the second output feature map groups is the target common divisor; and
obtaining the second output feature map groups corresponding to each first feature map group by performing the linear operation on the first feature map group for a plurality of times.

### Claim 8 (depends on 7)

The method of claim 7, wherein the obtaining the multi-channel first output feature maps by processing the multi-channel feature map through the pointwise convolution and the multi-channel second output feature maps by processing the multi-channel feature map through the non-pointwise convolution comprises:
obtaining the multi-channel first output feature maps by performing the pointwise convolution on the multi-channel feature maps; and
obtaining the multi-channel second output feature maps by performing the non-pointwise convolution on the multi-channel feature maps;
wherein the fusing the multi-channel first output feature maps and the multi-channel second output feature maps to obtain the multi-channel third output feature map comprises:
obtaining the multi-channel third output feature map by performing an accumulation operation on the multi-channel first output feature maps and the multi-channel second output feature maps.

### Claim 9 (depends on 7)

The method of claim 7, wherein the multi-channel feature maps are divided into a first part of the multi-channel feature maps and a second part of the multi-channel feature maps by channel, and the obtaining the multi-channel first output feature maps by processing the multi-channel feature map through the pointwise convolution and the multi-channel second output feature maps by processing the multi-channel feature map through the non-pointwise convolution comprises:
obtaining the multi-channel first output feature maps by performing the pointwise convolution on the first part of the multi-channel feature maps; and
obtaining the multi-channel second output feature maps by performing the non-pointwise convolution on the second part of the multi-channel feature maps;
wherein the fusing the multi-channel first output feature maps and the multi-channel second output feature maps to obtain the multi-channel third output feature map comprises:
obtaining the multi-channel third output feature map by performing a channel reorganization on the multi-channel first output feature maps and the multi-channel second output feature maps.