# US 12,112,486 B2 · Method and system for optimizing optical flow for images based on residual field and displacement field

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,486 B2
- Document ID: US12112486B2
- Status: Granted
- Kind: B2
- Application number: 17/570,836
- Filed: 2022-01-07
- Published: 2022-07-14
- Granted: 2024-10-08
- Earliest claimed: 2021-01-08
- Assignees: Nanchang Hangkong University
- Applicants: Nanchang Hangkong University
- Inventors: Congxuan Zhang; Cheng Feng; Zhen Chen; Weiming Hu; Ming Li; Hao Chen
- Web page: https://patentgrep.com/patent/US12112486B2

## CPC

- G06T 7/248: {involving reference images or patches}
- G06T 7/269: using gradient-based methods
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- Y02T 10/40: Engine management systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

A method and system for optimizing optical flow for images based on a residual field and a displacement field are provided. The method includes: obtaining reference images; estimating an initial optical flow field from the reference images by using an optical flow estimation method; obtaining an optical flow optimization model, where the optical flow optimization model includes an image encoder, an optical flow encoder, a first decoder, and a sub-decoder; inputting any image of the reference images and the initial optical flow field into the optical flow optimization model to output the residual field and the displacement field; superimposing the initial optical flow field and the residual field to obtain a preliminarily optimized optical flow field; and resampling the preliminarily optimized optical flow field by using the displacement field to obtain an optimized optical flow field.

## Claims

### Claim 1

A method for optimizing optical flow for images based on a residual field and a displacement field, comprising:
obtaining reference images, the reference images are two adjacent images in an image sequence;
estimating an initial optical flow field from the reference images by using an optical flow estimation method;
obtaining an optical flow optimization model, wherein the optical flow optimization model comprises an image encoder, an optical flow encoder, a first decoder, and a sub-decoder;
inputting any image of the reference images and the initial optical flow field into the optical flow optimization model to output the residual field and the displacement field;
superimposing the initial optical flow field and the residual field to obtain a preliminarily optimized optical flow field; and
resampling the preliminarily optimized optical flow field by using the displacement field to obtain an optimized optical flow field.

### Claim 2 (depends on 1)

The method for optimizing the optical flow for the images based on the residual field and the displacement field according to claim 1, wherein the image encoder comprises a plurality of convolutional layers, the optical flow encoder comprises a plurality of convolutional layers, the first decoder comprises a plurality of convolutional layers, and the sub-decoder comprises a first sub-decoder and a second sub-decoder.

### Claim 3 (depends on 2)

The method for optimizing the optical flow for the images based on the residual field and the displacement field according to claim 2, wherein inputting the any image of the reference images and the initial optical flow field into the optical flow optimization model to output the residual field and the displacement field comprises:
performing down-sampling and layering of feature pyramid on the any image of the reference images by using the image encoder to obtain a plurality of image feature maps with different resolutions;
performing down-sampling and layering of feature pyramid on the initial optical flow field by using the optical flow encoder to obtain a plurality of optical flow field feature maps with different resolutions;
generating a decoded feature map by using the first decoder based on the plurality of image feature maps with different resolutions and the plurality of optical flow field feature maps with different resolutions;
calculating the residual field by using the first sub-decoder based on the decoded feature map; and
calculating the displacement field by using the second sub-decoder based on the decoded feature map.

### Claim 4 (depends on 3)

The method for optimizing the optical flow for the images based on the residual field and the displacement field according to claim 3, wherein generating the decoded feature map by using the first decoder based on the plurality of image feature maps with different resolutions and the plurality of optical flow field feature maps with different resolutions comprises:
generating the decoded feature map by using a formula Xd1=D1 (concatenate(Xd2+Xer2,Xef1,Xef2,Xef3))+Xer1, wherein the first decoder comprises four convolutional layers, D1 is a convolution operation of a first convolutional layer, Xd1 is the decoded feature map output by the first decoder, concatenate is a channel superposition operation, Xd2=D2 ((concatenate(Xd3+Xer3,Xef1,Xef2,Xef3)), D2 is a convolution operation of a second convolutional layer, Xd3=D3 (concatenate(Xd4+Xef1,Xef2,Xef3)), D3 is a convolution operation of a third convolutional layer, Xd4=D4(Xef3), D4 is a convolution operation of a fourth convolutional layer, Xer1, Xer2 and Xer3 are the plurality of image feature maps with different resolutions output by the image encoder, and Xef1, Xef3, and Xef3 are the plurality of optical flow field feature maps with different resolutions output by the optical flow encoder.

### Claim 5

A system for optimizing optical flow for images based on a residual field and a displacement field, comprising:
a reference image obtaining module, configured to obtain reference images, the reference images are two adjacent images in an image sequence;
an optical flow estimation module, configured to estimate an initial optical flow field from the reference images by using an optical flow estimation method;
an optical flow optimization model obtaining module, configured to obtain the optical flow optimization model, wherein the optical flow optimization model comprises an image encoder, an optical flow encoder, a first decoder, and a sub-decoder;
a residual field and displacement field calculation module, configured to input any image of the reference images and the initial optical flow field into the optical flow optimization model to output the residual field and the displacement field;
a superposition module, configured to superimpose the initial optical flow field and the residual field to obtain a preliminarily optimized optical flow field; and
a resampling module, configured to resample the preliminarily optimized optical flow field by using the displacement field to obtain an optimized optical flow field.

### Claim 6 (depends on 5)

The system for optimizing the optical flow for the images based on the residual field and the displacement field according to claim 5, wherein the image encoder comprises a plurality of convolutional layers, the optical flow encoder comprises a plurality of convolutional layers, the first decoder comprises a plurality of convolutional layers, and the sub-decoder comprises a first sub-decoder and a second sub-decoder.

### Claim 7 (depends on 6)

The system for optimizing the optical flow for the images based on the residual field and the displacement field according to claim 6, wherein the residual field and displacement field calculation module comprises:
an image feature extraction unit, configured to perform down-sampling and layering of feature pyramid on the any image of the reference images by using the image encoder to obtain a plurality of image feature maps with different resolutions;
an optical flow field feature extraction unit, configured to perform down-sampling and layering of feature pyramid on the initial optical flow field by using the optical flow encoder to obtain a plurality of optical flow field feature maps with different resolutions;
a first decoding unit, configured to generate a decoded feature map by using the first decoder based on the plurality of image feature maps with different resolutions and the plurality of optical flow field feature maps with different resolutions;
a residual field calculation unit, configured to calculate the residual field by using the first sub-decoder based on the decoded feature map; and
a displacement field calculation unit, configured to calculate the displacement field by using the second sub-decoder based on the decoded feature map.

### Claim 8 (depends on 7)

The system for optimizing the optical flow for the images based on the residual field and the displacement field according to claim 7, wherein the first decoding unit comprises:
a decoding subunit, configured to generate the decoded feature map by using a formula Xd1=D1 (concatenate(Xd2+Xer2,Xef1,Xef2,Xef3))+Xer1, wherein the first decoder comprises four convolutional layers, D1 is a convolution operation of a first convolutional layer, Xd1 is the decoded feature map output by the first decoder, concatenate is a channel superposition operation, Xd2=D2 (concatenate(Xd3+Xer3,Xef1,Xef2,Xef3)), D2 is a convolution operation of a second convolutional layer, Xd3=D3 (concatenate(Xd4+Xef1,Xef2,Xef3)) D3 is a convolution operation of a third convolutional layer, Xd4=D4(Xef3), D4 is a convolution operation of a fourth convolutional layer, Xer1, Xer2 and Xer3 are the plurality of image feature maps with different resolutions output by the image encoder, Xer1, Xef2 and Xef3 are the plurality of optical flow field feature maps with different resolutions output by the optical flow encoder.