# US 12,112,528 B2 · Method and apparatus for processing images, device and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,528 B2
- Document ID: US12112528B2
- Status: Granted
- Kind: B2
- Application number: 17/418,692
- Filed: 2019-12-26
- Published: 2022-03-17
- Granted: 2024-10-08
- Earliest claimed: 2018-12-29
- Assignees: Bigo Technology Pte. Ltd.
- Applicants: Bigo Technology Pte. Ltd.
- Inventors: Zhuanghui Zhang; Zhujin Liang; Jundong Wang; Depeng Liang; Shuye Zhang
- Web page: https://patentgrep.com/patent/US12112528B2

## CPC

- G06V 10/806: {of extracted features}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/253: {of extracted features}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for processing images, including: acquiring an original image and auxiliary information of the original image; acquiring an object feature map by inputting the original image into a main path of a first visual task processing model, and acquiring an auxiliary feature map by inputting the auxiliary information into a branch path of the first visual task processing model; and acquiring a response map of the original image by fusing the object feature map and the auxiliary feature map and inputting fused object feature map and auxiliary feature map into the main path of the first visual task processing model.

## Claims

### Claim 1

A method for processing images, comprising:
acquiring an original image and auxiliary information of the original image;
acquiring an object feature map by inputting the original image into a main path of a first visual task processing model, and acquiring an auxiliary feature map by inputting the auxiliary information into a branch path of the first visual task processing model; and
acquiring a response map of the original image by fusing the object feature map and the auxiliary feature map and inputting fused object feature map and auxiliary feature map into the main path of the first visual task processing model,
wherein in the case that the original image is a single image, the auxiliary information of the original image comprises a background image corresponding to the original image, and in the case that the original image is a video frame in a video, and the video frame is used as a current video frame which is not a first frame of the video, the auxiliary information of the original image comprises a previous video frame of the current video frame and a response map of the previous video frame,
wherein the response map of the previous video frame is acquired in following manners:
when the previous video frame belongs to one of first N video frames of a video, the response map of the previous video frame is a response map acquired by inputting the previous video frame into a second visual task processing model; and
when the previous video frame does not belong to one of the first N video frames of the video, the response map of the previous video frame is a response map acquired by inputting the previous video frame into the first visual task processing model,
wherein the second visual task processing model has higher prediction accuracy than the first visual task processing model, and N is a positive integer.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the response map of the previous video frame is acquired by following manners:
when the previous video frame belongs to one of first T video frames of a video frame sequence corresponding to the previous video frame, the response map of the previous video frame is a response map acquired by inputting the previous video frame into a second visual task processing model; and
when the previous video frame does not belong to one of the first T video frames of the video frame sequence corresponding to the previous video frame, the response map of the previous video frame is a response map acquired by inputting the previous video frame into the first visual task processing model,
wherein the video frame sequence is one of a plurality of video frame sequences acquired by dividing a plurality of video frames of a video, the second visual task processing model has higher prediction accuracy than the first visual task processing model, and T is a positive integer.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the first visual task processing model is trained in following manners:
acquiring an original training image, annotation information of the original training image and auxiliary training information of the original training image;
acquiring an object training feature map by inputting the original training image into a main path of a convolutional neural network, and acquiring an auxiliary training feature map by inputting the auxiliary training information into a branch path of the convolutional neural network;
acquiring a response map of the original training image by fusing the object training feature map and the auxiliary training feature map and inputting fused object training feature map and auxiliary training feature map into the main path of the convolutional neural network;
acquiring a loss function of the convolutional neural network according to the annotation information of the original training image and the response map of the original training image; and
adjusting a network parameter of the convolutional neural network according to the loss function until an output value of the loss function is less than or equal to a preset threshold, and taking the convolutional neural network as the first visual task processing model.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the auxiliary training information is auxiliary training information acquired after being subjected to a data augmenting processing.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the data augmenting processing comprises at least one of translation, rotation, clipping, non-rigid transformation, noise interference and color transformation.

### Claim 6

An apparatus for processing images, comprising:
an original image and auxiliary information acquiring module configured to acquire an original image and auxiliary information of the original image;
a feature map acquiring module configured to acquire an object feature map by inputting the original image into a main path of a first visual task processing model, and acquire an auxiliary feature map by inputting the auxiliary information into a branch path of the first visual task processing model; and
an original image response map acquiring module configured to acquire a response map of the original image by fusing the object feature map and the auxiliary feature map and inputting fused object feature map and auxiliary feature map into the main path of the first visual task processing model;
wherein in the case that the original image is a single image, the auxiliary information of the original image comprises a background image corresponding to the original image, and in the case that the original image is a video frame in a video, and the video frame is used as a current video frame which is not a first frame of the video, the auxiliary information of the original image comprises a previous video frame of the current video frame and a response map of the previous video frame,
wherein the response map of the previous video frame is acquired in following manners:
when the previous video frame belongs to one of first N video frames of a video, the response map of the previous video frame is a response map acquired by inputting the previous video frame into a second visual task processing model; and
when the previous video frame does not belong to one of the first N video frames of the video, the response map of the previous video frame is a response map acquired by inputting the previous video frame into the first visual task processing model,
wherein the second visual task processing model has higher prediction accuracy than the first visual task processing model, and N is a positive integer.

### Claim 7 (depends on 1)

A device, comprising:
one or more processors; and
a memory configured to store one or more programs,
wherein the one or more programs is executed by the one or more processors to cause the one or more processors to realize the method according to claim 1.

### Claim 8 (depends on 1)

A non-transitory computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, the method according to claim 1 is realized.

### Claim 9 (depends on 2)

A device, comprising:
one or more processors; and
a memory configured to store one or more programs,
wherein the one or more programs is executed by the one or more processors to cause the one or more processors to realize the method according to claim 2.

### Claim 10 (depends on 2)

A non-transitory computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, the method according to claim 2 is realized.

### Claim 11 (depends on 3)

A device, comprising:
one or more processors; and
a memory configured to store one or more programs,
wherein the one or more programs is executed by the one or more processors to cause the one or more processors to realize the method according to claim 3, and wherein first visual task processing model is used to process a visual task, and the visual task includes one of image classification, image segmentation, object detection, key point positioning, and posture estimation.

### Claim 12 (depends on 4)

A device, comprising:
one or more processors; and
a memory configured to store one or more programs,
wherein the one or more programs is executed by the one or more processors to cause the one or more processors to realize the method according to claim 4.

### Claim 13 (depends on 5)

A device, comprising:
one or more processors; and
a memory configured to store one or more programs,
wherein the one or more programs is executed by the one or more processors to cause the one or more processors to realize the method according to claim 5.