# US 2022/0284552 A1 · Temporal feature alignment network for video inpainting

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284552 A1
- Document ID: US20220284552A1
- Status: Published application
- Kind: A1
- Application number: 17/192,549
- Filed: 2021-03-04
- Published: 2022-09-08
- Earliest claimed: 2021-03-04
- Assignees: Lemon Inc.
- Applicants: Lemon Inc.
- Inventors: Linjie Yang; Ding Liu; Xueyan Zou
- Web page: https://patentgrep.com/patent/US20220284552A1

## CPC

- G06T 5/80: Geometric correction
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 3/096: Transfer learning
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods are directed to inpainting video. More specifically, initial video data including a sequence of image frames containing missing or corrupted pixel information may be received. Optical flow displacement values and optical flow validity masks may be generated for neighboring image frames of initial video data. Image features from image feature maps of one or more neighboring image frames may be warp-shifted to image feature maps of a current image frame using the optical flow displacement values and warp-shifted image features from the feature maps of the one or more neighboring image frames may be selected based on one or more of the optical flow validity masks. A sequence of complete image frames may be generated based on the selected warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame.

## Claims

### Claim 1

A method for inpainting a sequence of initial image frames, the method comprising:
receiving initial video data representing a sequence of initial image frames;
generating optical flow displacement values between neighboring image frames of the sequence of initial image frames;
warp-shifting image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame using the optical flow displacement values; and
generating a sequence of complete image frames based on warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame, the sequence of complete image frames including an inpainted version of the sequence of initial image frames.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating optical flow validity masks for respective image frames of the sequence of image frames based on the optical flow displacement values; and
selecting warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame based on one or more of the optical flow validity masks,
wherein the sequence of complete image frames is based on the selected warp-shifted image features from the feature maps of the one or more neighboring image frames and the selected image features from the image feature maps of the current image frame.

### Claim 3 (depends on 1)

The method of claim 1, wherein warp-shifting image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame includes:
temporally shifting features from a first image feature map in a first channel of a previous image frame to the current image frame;
temporally shifting features from a second image feature map in a second channel of a subsequent image frame to the current image frame;
warping the shifted image features from the first image feature map in the first channel to a time consistent with the current image frame; and
warping the shifted image features from the first image feature map in the first channel to a time consistent with the current image frame.

### Claim 4 (depends on 2)

The method of claim 2, wherein generating optical flow validity masks for respective image frames of the sequence of image frames based on the optical flow displacement values includes:
mapping a first pixel location in a first image frame to a pixel location in a second image frame using a backward optical flow displacement value;
mapping the pixel location in the second image frame to a second pixel location in the first image frame using a forward optical flow displacement value; and
determining a distance between the first pixel location in the first image frame and the second pixel location in the first image frame.

### Claim 5 (depends on 4)

The method of claim 4, further comprising indicating that optical flow displacement values associated with the first pixel location in the first image frame are valid when the determined distance between the first pixel location in the first image frame and the second pixel location in the first image frame is less than a threshold distance value.

### Claim 6 (depends on 2)

The method of claim 2, further comprising:
providing the optical flow displacement values and the optical flow validity mask to a trained convolutional neural network model including a plurality of encoding layers and a plurality of decoding layers; and
at one or more encoding layers of the plurality of encoding layers and one or more decoding layers of the plurality of decoding layers:
warp-shifting image features from an image feature map of a first channel of a first neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value;
warp-shifting image features from an image feature map of a second channel of a second neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value; and
concatenating the feature maps along the channel dimension.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
applying a convolution operation to the concatenated feature maps; and
combining the results of the convolution operation applied to the concatenated feature maps with a gating signal.

### Claim 8

A system configured to perform video inpainting, the system comprising:
a processor; and
memory including instructions, which when executed by the processor, causes the processor to:
receive initial video data representing a sequence of initial image frames;
generate optical flow displacement values between neighboring image frames of the sequence of initial image frames;
at one or more encoding layers and at one or more decoding layers of a trained convolutional neural network, warp-shift image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame using the optical flow displacement values; and
generate a sequence of complete image frames based on warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame, the sequence of complete image frames including an inpainted version of the sequence of initial image frames.

### Claim 9 (depends on 8)

The system of claim 8, wherein the instructions, when executed by the processor, cause the processor to:
generate optical flow validity masks for respective image frames of the sequence of image frames based on the optical flow displacement values; and
select warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame based on one or more of the optical flow validity masks,
wherein the sequence of complete image frames is based on the selected warp-shifted image features from the feature maps of the one or more neighboring image frames and the selected image features from the image feature maps of the current image frame.

### Claim 10 (depends on 9)

The system of claim 9, wherein the instructions, when executed by the processor, cause the processor to:
temporally shift features from a first image feature map in a first channel of a previous image frame to the current image frame;
temporally shift features from a second image feature map in a second channel of a subsequent image frame to the current image frame;
warp the shifted image features from the first image feature map in the first channel to a time consistent with the current image frame; and
warp the shifted image features from the first image feature map in the first channel to a time consistent with the current image frame.

### Claim 11 (depends on 9)

The system of claim 9, wherein the instructions, when executed by the processor, cause the processor to:
map a first pixel location in a first image frame to a pixel location in a second image frame using a backward optical flow displacement value;
map the pixel location in the second image frame to a second pixel location in the first image frame using a forward optical flow displacement value; and
determine a distance between the first pixel location in the first image frame and the second pixel location in the first image frame.

### Claim 12 (depends on 11)

The system of claim 11, wherein the instructions, when executed by the processor, cause the processor to indicate that optical flow displacement values associated with the first pixel location in the first image frame are valid when the determined distance between the first pixel location in the first image frame and the second pixel location in the first image frame is less than a threshold distance value.

### Claim 13 (depends on 9)

The system of claim 9, wherein the instructions, when executed by the processor, cause the processor to:
provide the optical flow displacement values and the optical flow validity mask to the trained convolutional neural network; and
at one or more encoding layers of the plurality of encoding layers and one or more decoding layers of the plurality of decoding layers:
warp-shift image features from an image feature map of a first channel of a first neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value;
warp-shift image features from an image feature map of a second channel of a second neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value; and
concatenate the feature maps along the channel dimension.

### Claim 14 (depends on 13)

The system of claim 13, wherein the instructions, when executed by the processor, cause the processor to:
apply a convolution operation to the concatenated feature maps; and
combine the results of the convolution operation applied to the concatenated feature maps with a gating signal.

### Claim 15 (depends on 11)

The system of claim 11, wherein the instructions, when executed by the processor, cause the processor to provide the sequence of complete image frames to a computing system.

### Claim 16

A computer-readable storage medium including instructions, which when executed by a processor, cause the processor to:
receive initial video data representing a sequence of initial image frames;
generate optical flow displacement values between neighboring image frames of the sequence of initial image frames;
generate optical flow validity masks for respective image frames of the sequence of image frames based on the optical flow displacement values;
warp-shift image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame using the optical flow displacement values;
select warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame based on one or more of the optical flow validity masks; and
generate a sequence of complete image frames based on the selected warp-shifted image features from the feature maps of the one or more neighboring image frames and the selected image features from the image feature maps of the current image frame, the sequence of complete image frames including an inpainted version of the sequence of initial image frames.

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein warp-shifting image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame includes:
temporally shifting features from a first image feature map in a first channel of a previous image frame to the current image frame;
temporally shifting features from a second image feature map in a second channel of a subsequent image frame to the current image frame;
warping the shifted image features from the first image feature map in the first channel to a time consistent with the current image frame; and
warping the shifted image features from the first image feature map in the first channel to a time consistent with the current image frame.

### Claim 18 (depends on 16)

The computer-readable storage medium of claim 16, wherein generating optical flow validity masks for respective image frames of the sequence of image frames based on the optical flow displacement values includes:
mapping a first pixel location in a first image frame to a pixel location in a second image frame using a backward optical flow displacement value;
mapping the pixel location in the second image frame to a second pixel location in the first image frame using a forward optical flow displacement value; and
determining a distance between the first pixel location in the first image frame and the second pixel location in the first image frame.

### Claim 19 (depends on 18)

The computer-readable storage medium of claim 18, further comprising instructions, which when executed by the processor, cause the processor to indicate that optical flow displacement values associated with the first pixel location in the first image frame are valid when the determined distance between the first pixel location in the first image frame and the second pixel location in the first image frame is less than a threshold distance value.

### Claim 20 (depends on 18)

The computer-readable storage medium of claim 18, further comprising instructions, which when executed by the processor, cause the processor to:
provide the optical flow displacement values and the optical flow validity mask to a trained convolutional neural network model including a plurality of encoding layers and a plurality of decoding layers;
warp-shift image features from an image feature map of a first channel of a first neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value at one or more encoding layers of the plurality of encoding layers and one or more decoding layers of the plurality of decoding layers;
warp-shift image features from an image feature map of a second channel of a second neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value at one or more encoding layers of the plurality of encoding layers and one or more decoding layers of the plurality of decoding layers; and
concatenate the feature maps along the channel direction.
