# US 2021/0334579 A1 · Method and apparatus for processing video frame

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334579 A1
- Document ID: US20210334579A1
- Status: Published application
- Kind: A1
- Application number: 17/184,379
- Filed: 2021-02-24
- Published: 2021-10-28
- Earliest claimed: 2020-04-22
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science and Technology Co., Ltd.
- Inventors: Tianwei LIN; Xin Li; Fu Li; Dongliang He; Hao Sun; Henan Zhang
- Web page: https://patentgrep.com/patent/US20210334579A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06F 18/253: {of extracted features}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/094: Adversarial learning
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- G06T 2207/20216: Image averaging
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30168: Image quality inspection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and apparatus for processing a video frame are provided. The method may include: converting, using an optical flow generated based on a previous frame and a next frame of adjacent frames in a video, a feature map of the previous frame to obtain a converted feature map; determining, based on an error of the optical flow, a weight of the converted feature map, and obtaining a fused feature map based on a weighted result of a feature of the converted feature map and a feature of a feature map of the next frame; and updating the feature map of the next frame as the fused feature map.

## Claims

### Claim 1

A method for processing a video frame, comprising:
converting, using an optical flow generated based on a previous frame and a next frame of adjacent frames in a video, a feature map of the previous frame to obtain a converted feature map;
determining, based on an error of the optical flow, a weight of the converted feature map, and obtaining a fused feature map based on a weighted result of a feature of the converted feature map and a feature of a feature map of the next frame; and
updating the feature map of the next frame to obtain an updated feature map, the updated feature map being the fused feature map.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the optical flow is a dense optical flow; and
the method further comprises:
contracting the previous frame to a size of the feature map of the previous frame to obtain a contracted previous frame;
contracting the next frame to a size of the feature map of the next frame to obtain a contracted next frame;
determining a dense optical flow between the contracted previous frame and the contracted next frame; and
using the dense optical flow as the optical flow generated based on the previous frame and the next frame of the adjacent frames in the video.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the determining, based on the error of the optical flow, the weight of the converted feature map comprises:
converting the contracted previous frame using the dense optical flow to obtain a converted previous frame;
determining, based on a difference between a pixel value of a pixel at each coordinate in the converted previous frame and a pixel value of the pixel at the coordinate in the contracted next frame, the error of the dense optical flow of the pixel at the coordinate;
determining, for a pixel at the coordinate of an image obtained by converting a specified image using the dense optical flow, whether there is a pixel including a same object in the specified image, to obtain a determination result, wherein a size of the specified image is a target size; and
determining a weight of the pixel at the coordinate in the converted feature map based on the error of the dense optical flow and the determination result.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the determining, for the pixel at the coordinate of the image obtained by converting the specified image using the dense optical flow, whether there is the pixel including the same object in the specified image, to obtain the determination result, wherein the size of the specified image is the target size comprises:
acquiring the specified image with a pixel value of each pixel being a preset pixel value;
converting the specified image using the dense optical flow to obtain a converted specified image;
determining, for the pixel at each coordinate in the converted specified image, whether a pixel value of the pixel at the coordinate in the converted specified image is greater than or equal to the preset pixel value;
determining, in response to the pixel value of the pixel at the coordinate in the converted specified image being greater than or equal to the preset pixel value, the determination result being that there is the pixel including the same object in the specified image; and
determining, in response to the pixel value of the pixel at the coordinate in the converted specified image being less than the preset pixel value, the determination result being that there is no pixel including the same object in the specified image.

### Claim 5 (depends on 3)

The method according to claim 3, wherein the determining the weight of the pixel at the coordinate in the converted feature map based on the error of the dense optical flow and the determination result comprises:
determining, in response to determining that the error of the dense optical flow is less than a specified error threshold, and the determination result is that there is the pixel including the same object in the specified image, that the weight of the pixel at the coordinate in the converted feature map is a first candidate weight, wherein the greater the pixel value of the pixel at the coordinate in the contracted next frame is, the higher the specified error threshold is; and
determining, in response to determining that the error of the dense optical flow is more than or equal to the specified error threshold, and/or the determination result is that there is no pixel including the same object in the specified image, that the weight of the pixel at the coordinate in the converted feature map is a second candidate weight, wherein the first candidate weight is higher than the second candidate weight.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the method further comprises:
weighting the feature of the converted feature map and the feature of the feature map of the next frame based on the weight of the converted feature map and a weight of the feature map of the next frame to obtain the weighted result of the feature of the converted feature map and the feature of the feature map of the next frame, wherein the higher the weight of the converted feature map is, the lower the weight of the feature map of the next frame is.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the method further comprises:
generating the feature map of the next frame and the feature map of the previous frame of the video using a generative adversarial network; and
the method further comprises:
processing the updated feature map using the generative adversarial network, to generate an image of a target domain corresponding to the next frame.

### Claim 8

An electronic device, comprising:
one or more processors; and
a storage apparatus for storing one or more programs,
the one or more programs, when executed by the one or more processors, causing the one or more processors to perform operations comprising:
converting, using an optical flow generated based on a previous frame and a next frame of adjacent frames in a video, a feature map of the previous frame to obtain a converted feature map;
determining, based on an error of the optical flow, a weight of the converted feature map, and obtaining a fused feature map based on a weighted result of a feature of the converted feature map and a feature of a feature map of the next frame; and
updating the feature map of the next frame to obtain an updated feature map, the updated feature map being the fused feature map.

### Claim 9 (depends on 8)

The electronic device according to claim 8, wherein the optical flow is a dense optical flow; and
the operations further comprise:
contracting the previous frame to a size of the feature map of the previous frame to obtain a contracted previous frame;
contracting the next frame to a size of the feature map of the next frame to obtain a contracted next frame;
determining a dense optical flow between the contracted previous frame and the contracted next frame; and
using the dense optical flow as the optical flow generated based on the previous frame and the next frame of the adjacent frames in the video.

### Claim 10 (depends on 9)

The electronic device according to claim 9, wherein the determining, based on the error of the optical flow, the weight of the converted feature map comprises:
converting the contracted previous frame using the dense optical flow to obtain a converted previous frame;
determining, based on a difference between a pixel value of a pixel at each coordinate in the converted previous frame and a pixel value of the pixel at the coordinate in the contracted next frame, the error of the dense optical flow of the pixel at the coordinate;
determining, for a pixel at the coordinate of an image obtained by converting a specified image using the dense optical flow, whether there is a pixel including a same object in the specified image, to obtain a determination result, wherein a size of the specified image is a target size; and
determining a weight of the pixel at the coordinate in the converted feature map based on the error of the dense optical flow and the determination result.

### Claim 11 (depends on 10)

The electronic device according to claim 10, wherein the determining, for the pixel at the coordinate of the image obtained by converting the specified image using the dense optical flow, whether there is the pixel including the same object in the specified image, to obtain the determination result, wherein the size of the specified image is the target size comprises:
acquiring the specified image with a pixel value of each pixel being a preset pixel value;
converting the specified image using the dense optical flow to obtain a converted specified image;
determining, for the pixel at each coordinate in the converted specified image, whether a pixel value of the pixel at the coordinate in the converted specified image is greater than or equal to the preset pixel value;
determining, in response to the pixel value of the pixel at the coordinate in the converted specified image being greater than or equal to the preset pixel value, the determination result being that there is the pixel including the same object in the specified image; and
determining, in response to the pixel value of the pixel at the coordinate in the converted specified image being less than the preset pixel value, the determination result being that there is no pixel including the same object in the specified image.

### Claim 12 (depends on 10)

The electronic device according to claim 10, wherein the determining the weight of the pixel at the coordinate in the converted feature map based on the error of the dense optical flow and the determination result comprises:
determining, in response to determining that the error of the dense optical flow is less than a specified error threshold, and the determination result is that there is the pixel including the same object in the specified image, that the weight of the pixel at the coordinate in the converted feature map is a first candidate weight, wherein the greater the pixel value of the pixel at the coordinate in the contracted next frame is, the higher the specified error threshold is; and
determining, in response to determining that the error of the dense optical flow is more than or equal to the specified error threshold, and/or the determination result is that there is no pixel including the same object in the specified image, that the weight of the pixel at the coordinate in the converted feature map is a second candidate weight, wherein the first candidate weight is higher than the second candidate weight.

### Claim 13 (depends on 8)

The electronic device according to claim 8, wherein the operations further comprise:
weighting the feature of the converted feature map and the feature of the feature map of the next frame based on the weight of the converted feature map and a weight of the feature map of the next frame to obtain the weighted result of the feature of the converted feature map and the feature of the feature map of the next frame, wherein the higher the weight of the converted feature map is, the lower the weight of the feature map of the next frame is.

### Claim 14 (depends on 8)

The electronic device according to claim 8, wherein the operations further comprise:
generating the feature map of the next frame and the feature map of the previous frame of the video using a generative adversarial network; and
the operations further comprise:
processing the updated feature map using the generative adversarial network, to generate an image of a target domain corresponding to the next frame.

### Claim 15

A non-transitory computer readable storage medium, storing a computer program thereon, the computer program, when executed by a processor, causing the processor to perform operations comprising:
converting, using an optical flow generated based on a previous frame and a next frame of adjacent frames in a video, a feature map of the previous frame to obtain a converted feature map;
determining, based on an error of the optical flow, a weight of the converted feature map, and obtaining a fused feature map based on a weighted result of a feature of the converted feature map and a feature of a feature map of the next frame; and
updating the feature map of the next frame, the updated feature map being the fused feature map.
