Temporal feature alignment network for video inpainting

A system configured to perform video inpainting, the system comprising:

• a processor;
• and
• memory including instructions, which when executed by the processor, causes the processor to:
• receive initial video data representing a sequence of initial image frames;
• generate optical flow displacement values between neighboring image frames of the sequence of initial image frames;
• at one or more encoding layers and at one or more decoding layers of a trained convolutional neural network, warp-shift image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame using the optical flow displacement values;
• and
• generate a sequence of complete image frames based on warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame, the sequence of complete image frames including an inpainted version of the sequence of initial image frames.

A computer-readable storage medium including instructions, which when executed by a processor, cause the processor to:

• receive initial video data representing a sequence of initial image frames;
• generate optical flow displacement values between neighboring image frames of the sequence of initial image frames;
• generate optical flow validity masks for respective image frames of the sequence of image frames based on the optical flow displacement values;
• warp-shift image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame using the optical flow displacement values;
• select warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame based on one or more of the optical flow validity masks;
• and
• generate a sequence of complete image frames based on the selected warp-shifted image features from the feature maps of the one or more neighboring image frames and the selected image features from the image feature maps of the current image frame, the sequence of complete image frames including an inpainted version of the sequence of initial image frames.