Matte-based video restoration
AssigneeMicrosoft Corporation
InventorsSing Bing Kang, Jiading Gai
Life of this patent
Estimating the term…
Abstract
Matte-based video restoration technique embodiments are presented which model spatio-temporally varying film wear artifacts found in digitized copies of film media. In general, this is accomplished by employing residual color information in recovering of artifact mattes. To this end, the distributions of artifact colors and their fractional contribution to each pixel of each frame being considered are extracted based on color information from the spatial and temporal neighborhoods of the pixel. The extracted information can then be used to restore the video by removing the artifacts.
Claims20
A computer-implemented process for restoring frames of a video derived from a film-based medium by removing spatio-temporally varying artifacts caused by film wear, the process comprising using a computer to perform the following process actions:
- inputting a prescribed number of consecutive frames of the video;
- estimating a distribution of prescribed artifact colors in the pixels of the inputted frames, and for each pixel in the inputted frames, estimating an alpha value which represents the fractional contribution to the pixel of a true color the pixel would exhibit without any artifact color contamination, and estimating an alpha-premultiplied true color, said estimations being based on the colors of pixels observed in a prescribed spatial neighborhood and a prescribed temporal neighborhood of each pixel of the inputted frames and a prescribed initial value of the alpha-premultiplied true color for each pixel;
- repeating the estimating actions using the last-estimated alpha premultiplied true color of each pixel in lieu of the initial or last-employed alpha premultiplied true color, for a prescribed number of iterations, or until the last-estimated alpha premultiplied true color of each pixel does not change between iterations to more than a prescribed degree;
- estimating a restored color for each pixel of each inputted frame using the estimated alpha values and estimated alpha-premultiplied true colors associated with that pixel and the pixels in the prescribed spatial and temporal neighborhoods of that pixel;
- and
- replacing the observed color of each pixel in each inputted frame with the restored color for that pixel.
A computer-implemented process for restoring frames of a video derived from a film-based medium by removing spatio-temporally varying artifacts caused by film wear, the process comprising using a computer to perform the following process actions:
- (a) inputting a prescribed number of consecutive frames of the video, wherein the frames exhibit inter-frame motion in a scene depicted therein in excess of a prescribed threshold;
- (b) estimating the motion between each pair of consecutive frames based on observed colors of the pixels of the frames, and using the motion estimates to identify the locations of each pair of corresponding pixels between each pair of consecutive frames;
- (c) estimating a distribution of prescribed artifact colors in the pixels of the inputted frames, and for each pixel in the inputted frames, estimating an alpha value which represents the fractional contribution to the pixel of a true color the pixel would exhibit without any artifact color contamination, and estimating an alpha-premultiplied true color, said estimations being based on the colors of pixels observed in a prescribed spatial neighborhood and a prescribed temporal neighborhood of each pixel of the inputted frames and a prescribed initial value of the alpha-premultiplied true color for each pixel;
- (d) repeating the estimating actions of (c) using the last-estimated alpha premultiplied true color of each pixel in lieu of the initial or last-employed alpha premultiplied true color, for a prescribed number of iterations, or until the last-estimated alpha premultiplied true color of each pixel does not change between iterations to more than a prescribed degree;
- (e) estimating a current restored color for each pixel of each inputted frame using the estimated alpha values and estimated alpha-premultiplied true colors associated with that pixel and the pixels in the prescribed spatial and temporal neighborhoods of that pixel;
- (f) replacing the current color of each pixel in each inputted frame with the current restored color for that pixel;
- (g) repeating actions (b) through (f) using the current restored color for each pixel in place of the observed color or the current color, for a prescribed number of iterations, or until the last motion estimate for each pixel falls below a prescribed motion threshold;
- and
- (h) designating the current restored color for each pixel in each frame to be the final restored color.
A computer-implemented process for adding spatio-temporally varying artifacts to frames of a video to simulate artifacts caused by film wear in a film-based medium, the process comprising using a computer to perform the following process actions:
- inputting a prescribed number of consecutive frames of the video derived from a film-based medium that exhibited said spatio-temporally varying artifacts caused by film wear;
- estimating a distribution of prescribed artifact colors in the pixels of the inputted frames, and for each pixel in the inputted frames, estimating an alpha value which represents the fractional contribution to the pixel of a true color the pixel would exhibit without any artifact color contamination, and estimating an alpha-premultiplied true color, said estimations being based on the colors of pixels observed in a prescribed spatial neighborhood and a prescribed temporal neighborhood of each pixel of the inputted frames and a prescribed initial value of the alpha-premultiplied true color for each pixel;
- repeating the estimating actions using the last-estimated alpha premultiplied true color of each pixel in lieu of the initial or last-employed alpha premultiplied true color, for a prescribed number of iterations, or until the last-estimated alpha premultiplied true color of each pixel does not change between iterations to more than a prescribed degree;
- inputting a number of consecutive frames of a video not exhibiting spatio-temporally varying artifacts caused by film wear;
- and
- blending the last-estimated distribution of artifact colors into pixel locations of the inputted frames of the video not exhibiting spatio-temporally varying artifacts using the last-estimated alpha values.
Dependencies are inferred from the claim text.