Back to results
US 12,182,967 B2Granted
Super-resolution using natural handheld-motion applied to a user device
AssigneeGoogle Inc.
InventorsYi Hung Chen, Chia-Kai Liang, Bartlomiej Maciej Wronski, Peyman Milanfar, Ignacio Garcia Dorado
Life of this patent
Estimating the term…
Abstract
The present disclosure describes systems and techniques for creating a super-resolution image (122) of a scene captured by a user device (102). Natural handheld motion (110) introduces, across multiple frames (204, 206, 208) of an image of a scene, sub-pixel offsets that enable the use of super-resolution computations (210) to form color planes (212, 214, 216), which are accumulated (218) and combined (220) to create a super-resolution image (122) of the scene.
Claims20
A method used to render a super-resolution image of a scene, the method performed by a user device and comprising:
- capturing, in a burst sequence, multiple frames of an image of a scene, the multiple frames having respective, relative sub-pixel offsets of the image due to a motion of the user device during the capturing of the multiple frames;
- performing super-resolution computations using the captured, multiple frames, the super-resolution computations including robustness model computations;
- determining, contributions of pixels of the multiple frames to color planes, wherein the contributions of pixels of the multiple frames to the color planes are based on weighted summations of (i) samples of the pixels in local neighborhoods and (ii) a local robustness factor, wherein the local robustness factor is based on the robustness model computations;
- accumulating, based on the determined contributions of the pixels to the color planes, the color planes;
- combining the accumulated color planes to create the super-resolution image of the scene;
- and
- rendering the super-resolution image of the scene.
4 more elements
A method of providing color planes to an apparatus, the method comprising:
- computing Gaussian radial basis function kernels, wherein computing the Gaussian radial basis function kernels includes:
- computing a reference frame;
- and
- computing a kernel covariance matrix based on analyzing local gradient structure tensors, the local gradient structure tensors corresponding to edges, corners, or textured areas of content included in the reference frame;
- computing a robustness model, wherein computing the robustness model includes using a statistical neighborhood model to compute a color mean and spatial standard deviation;
- determining, contributions of pixels of multiple frames to color planes, wherein the contributions of pixels of the multiple frames to the color planes are based on weighted summations of (i) samples of the pixels in local neighborhoods and (ii) a local robustness factor, wherein the local robustness factor is based on the robustness model computations;
- accumulating, based on the determined contributions of the pixels to the color planes, the color planes;
- and
- providing, to the apparatus, the accumulated color planes.
6 more elements
A user device, the user device comprising:
- one or more image sensors;
- one or more processors;
- a display;
- and
- a computer-readable medium comprising instructions of a super-resolution manager application that, when executed by the one or more processors, directs the user device to:
- capture, in a burst sequence using the one or more image sensors, multiple frames of an image of a scene, the multiple frames having respective, relative offsets of the image across the multiple frames;
- perform, using the one or more processors, super-resolution computations using the captured, multiple frames, the super-resolution computations including robustness model computations;
- determine, using the one or more processors, contributions of pixels of the multiple frames to color planes, wherein the contributions of pixels of the multiple frames to the color planes are based on weighted summations of (i) samples of the pixels in local neighborhoods and (ii) a local robustness factor, wherein the local robustness factor is based on the robustness model computations;
- accumulate, using the one or more processors and based on determined contributions of the pixels to the color planes, the color planes;
- combine, using the one or more processors, the accumulated color planes to create a super-resolution image of the scene;
- and
- render, using the display, the super-resolution image of the scene.
9 more elements
Dependencies are inferred from the claim text.