Video denoising using optical flow

A non-transitory computer-readable storage memory comprising stored instructions that are executable and, responsive to execution of the instructions by a computing device, the computing device performs operations comprising to:

• receive image frames of video content as a sequence of the image frames, one or more of the image frames including noise that corrupts the video content, and one of the image frames in the sequence selected as a reference frame;
• determine matching patches that match an image patch in the reference frame, the matching patches being determined from within a local neighborhood around the image patch in the reference frame;
• compute a noise estimate for previous image frames and for subsequent image frames relative to the reference frame in the sequence of the image frames, the noise estimate for a particular one of the image frames computed based on optical flow, and the noise estimate usable to determine a contribution of similar motion patches in the particular image frame to denoise the image patch in the reference frame;
• adaptively select a number of the image frames in the sequence of the image frames as a temporal window of the previous and subsequent image frames relative to the reference frame to denoise the image patch in the reference frame, the number of the image frames being selected based on the computed noise estimate for each of the previous and subsequent image frames;
• determine the similar motion patches from the previous and subsequent image frames that correspond to the image patch in the reference frame, the similar motion patches being determined based on the optical flow computations;
• and
• denoise the image patch in the reference frame based on the matching patches that are determined from within the reference frame and based on the similar motion patches that are determined from the previous and subsequent image frames in the temporal window, the previous and subsequent image frames being used based on the noise estimates that are computed for each of the previous and subsequent image frames.

A system, comprising:

• a memory configured to store image frames of video content as a sequence of the image frames, one or more of the image frames including noise that corrupts the video content, and one of the image frames in the sequence selected as a reference frame;
• a processor system to implement a video denoising application that is configured to:
• determine matching patches that match an image patch in the reference frame, the matching patches being determined from within a local neighborhood around the image patch in the reference frame;
• select a number of the image frames in the sequence of the image frames as a temporal window of previous image frames and subsequent image frames relative to the reference frame in the sequence of the image frames for denoising the reference frame, the previous and subsequent image frames including similar motion patches to the image patch in the reference frame;
• warp one or more of the previous and subsequent image frames effective to orient the similar motion patches relative to the image patch of the reference frame for pixel-wise patch comparisons of the similar motion patches to the image patch, the one or more previous and subsequent image frames being warped based on optical flow computed between the reference frame and the previous image frames, and computed between the image frame and the subsequent image frames;
• compute a noise estimate for each of the previous and subsequent image frames, the noise estimate for a particular one of the image frames computed based on the optical flow computations, and the noise estimate usable to determine a contribution of the similar motion patches in the particular image frame to denoise the image patch in the reference frame;
• determine the similar motion patches from the previous and subsequent image frames that correspond to the image patch in the reference frame, the similar motion patches being determined based on the optical flow computations;
• and
• denoise the image patch in the reference frame based on the matching patches that are determined from within the reference frame and based on the similar motion patches that are determined from the previous and subsequent image frames in the temporal window, the previous and subsequent image frames being used based on the noise estimates that are computed for each of the previous and subsequent image frames.