Realtime video super resolution using motion vectors
AssigneeMicrosoft Technology Licensing, LLC
InventorsSushanth Rajasankar, Rafael V. Cintron, Frank Kam-Chuen Li
Life of this patent
Estimating the term…
Abstract
A low resolution image frame is received. A past reference low resolution image frame and its super resolution result are retained. Motion vectors between the input low resolution image frame and reference low resolution frame are computed. For each pixel in the low resolution image frame, a reference pixel is determined based on the computed motion vectors and the reference low resolution frame. If the reference pixel equals the corresponding input pixel, the reference super resolution result pixel is rendered for this pixel, and otherwise a corresponding pixel from the current input frame is rendered.
Claims20
A method of generating super resolution images using a machine learning model and a graphics processing unit, the method comprising:
- receiving, by a computing system, a low resolution image frame as a current input frame, a reference super resolution result, and a reference low resolution frame, wherein the reference super resolution result is updated by the machine learning model;
- computing, using the graphics processing unit, motion vectors between the current input frame and the reference low resolution frame;
- and
- for an input pixel in the current input frame:
- determining a position of a reference pixel based on the computed motion vectors computed using the graphics processing unit;
- determining that the position of the reference pixel is within a distance threshold of a position of a corresponding input pixel;
- and
- based on the determining that the position of the reference pixel is within the distance threshold, rendering a reference super resolution result pixel from the reference super resolution result associated with the input pixel in the current input frame, and otherwise rendering the input pixel in the current input frame.
A computing system, comprising:
- one or more processors including a graphics processing unit;
- and
- a computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by the processor, cause the computing system to perform operations comprising:
- receiving a low resolution image frame, a reference super resolution result, and a reference low resolution frame, wherein the reference super resolution result is updated by a machine learning model;
- computing, using the graphics processing unit, motion vectors between the low resolution image frame and the reference low resolution frame;
- and
- for each pixel in the low resolution image frame:
- generating a reference pixel based on the computed motion vectors computed using the graphics processing unit and the reference low resolution frame;
- and
- when the reference pixel is within a position threshold of a corresponding input pixel, rendering a reference super resolution result pixel for a current pixel, and otherwise rendering a corresponding pixel from the low resolution image frame.
A non-transitory computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by a processor of a computing system, cause the computing system to perform operations comprising:
- receiving a low resolution image frame as a current input frame, a reference super resolution result, and a reference low resolution frame, wherein the reference super resolution result is updated by a machine learning model;
- computing, using a graphics processing unit, motion vectors between the low resolution image frame and the reference low resolution frame;
- and
- for each pixel in the low resolution image frame:
- computing a reference pixel based on the computed motion vectors computed using the graphics processing unit and the reference low resolution frame;
- and
- when the reference pixel is within a threshold of a corresponding input pixel, rendering a reference super resolution result pixel for each pixel, and otherwise rendering a corresponding pixel from the current input frame.
Dependencies are inferred from the claim text.