# US 12,182,967 B2 · Super-resolution using natural handheld-motion applied to a user device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,967 B2
- Document ID: US12182967B2
- Status: Granted
- Kind: B2
- Application number: 17/263,814
- Filed: 2019-08-06
- Published: 2021-09-30
- Granted: 2024-12-31
- Earliest claimed: 2018-08-09
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Yi Hung Chen; Chia-Kai Liang; Bartlomiej Maciej Wronski; Peyman Milanfar; Ignacio Garcia Dorado
- Web page: https://patentgrep.com/patent/US12182967B2

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 3/4069: by subpixel displacements
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/337: {involving reference images or patches}
- G06T 2207/10024: Color image
- G06T 2207/20024: Filtering details
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20201: Motion blur correction
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure describes systems and techniques for creating a super-resolution image (122) of a scene captured by a user device (102). Natural handheld motion (110) introduces, across multiple frames (204, 206, 208) of an image of a scene, sub-pixel offsets that enable the use of super-resolution computations (210) to form color planes (212, 214, 216), which are accumulated (218) and combined (220) to create a super-resolution image (122) of the scene.

## Claims

### Claim 1

A method used to render a super-resolution image of a scene, the method performed by a user device and comprising:
capturing, in a burst sequence, multiple frames of an image of a scene, the multiple frames having respective, relative sub-pixel offsets of the image due to a motion of the user device during the capturing of the multiple frames;
performing super-resolution computations using the captured, multiple frames, the super-resolution computations including robustness model computations;
determining, contributions of pixels of the multiple frames to color planes, wherein the contributions of pixels of the multiple frames to the color planes are based on weighted summations of (i) samples of the pixels in local neighborhoods and (ii) a local robustness factor, wherein the local robustness factor is based on the robustness model computations;
accumulating, based on the determined contributions of the pixels to the color planes, the color planes;
combining the accumulated color planes to create the super-resolution image of the scene; and
rendering the super-resolution image of the scene.

### Claim 2 (depends on 1)

The method as recited by claim 1, wherein performing the super-resolution computations includes Gaussian radial basis function kernel computations.

### Claim 3 (depends on 1)

The method as recited by claim 1, wherein the motion of the user device corresponds to a natural handheld-motion made by a user of the user device during the burst sequence.

### Claim 4 (depends on 1)

The method as recited by claim 1, wherein performing the super-resolution computations includes filtering pixel signals from each of the multiple frames to generate color-specific image planes for respective color channels.

### Claim 5 (depends on 4)

The method as recited by claim 4, wherein performing the super-resolution computations includes aligning the color-specific image planes to a reference frame.

### Claim 6 (depends on 4)

The method as recited by claim 4, wherein the respective color channels correspond to a red color channel, a blue color channel, and a green color channel.

### Claim 7 (depends on 2)

The method as recited by claim 2, wherein the Gaussian radial basis function kernel computations include computing a kernel covariance matrix based on analyzing local gradient structure tensors of color-specific image planes aligned to a reference frame.

### Claim 8 (depends on 7)

The method as recited by claim 7, wherein the local gradient structure tensors include local gradient structure tensors that correspond to edges included in the reference frame.

### Claim 9 (depends on 7)

The method as recited by claim 7, wherein the local gradient structure tensors include local gradient structure tensors that correspond to corners included in the reference frame.

### Claim 10 (depends on 7)

The method as recited by claim 7, wherein the local gradient structure tensors include local gradient structure tensors that correspond to textured areas included in the reference frame.

### Claim 11 (depends on 1)

The method as recited by claim 1, wherein the robustness model computations use a statistical neighborhood model to compute a spatial color standard deviation.

### Claim 12 (depends on 11)

The method as recited by claim 11, wherein the robustness model computations further uses the statistical neighborhood model to compute a mean difference.

### Claim 13

A method of providing color planes to an apparatus, the method comprising:
computing Gaussian radial basis function kernels, wherein computing the Gaussian radial basis function kernels includes:
computing a reference frame; and
computing a kernel covariance matrix based on analyzing local gradient structure tensors, the local gradient structure tensors corresponding to edges, corners, or textured areas of content included in the reference frame;
computing a robustness model, wherein computing the robustness model includes using a statistical neighborhood model to compute a color mean and spatial standard deviation;
determining, contributions of pixels of multiple frames to color planes, wherein the contributions of pixels of the multiple frames to the color planes are based on weighted summations of (i) samples of the pixels in local neighborhoods and (ii) a local robustness factor, wherein the local robustness factor is based on the robustness model computations;
accumulating, based on the determined contributions of the pixels to the color planes, the color planes; and
providing, to the apparatus, the accumulated color planes.

### Claim 14 (depends on 13)

The method as recited by claim 13, wherein providing the accumulated color planes includes providing the accumulated color planes to the apparatus to store.

### Claim 15 (depends on 13)

The method as recited by claim 13, wherein providing the accumulated color planes to the apparatus includes providing the accumulated color planes to the apparatus to combine and render a super-resolution image.

### Claim 16

A user device, the user device comprising:
one or more image sensors;
one or more processors;
a display; and
a computer-readable medium comprising instructions of a super-resolution manager application that, when executed by the one or more processors, directs the user device to:
capture, in a burst sequence using the one or more image sensors, multiple frames of an image of a scene, the multiple frames having respective, relative offsets of the image across the multiple frames;
perform, using the one or more processors, super-resolution computations using the captured, multiple frames, the super-resolution computations including robustness model computations;
determine, using the one or more processors, contributions of pixels of the multiple frames to color planes, wherein the contributions of pixels of the multiple frames to the color planes are based on weighted summations of (i) samples of the pixels in local neighborhoods and (ii) a local robustness factor, wherein the local robustness factor is based on the robustness model computations;
accumulate, using the one or more processors and based on determined contributions of the pixels to the color planes, the color planes;
combine, using the one or more processors, the accumulated color planes to create a super-resolution image of the scene; and
render, using the display, the super-resolution image of the scene.

### Claim 17 (depends on 16)

The user device as recited by claim 16, wherein the one or more processors are multiple, different processors and include a central processing unit, an image processing unit, a digital signal processor, or a graphics processing unit.

### Claim 18 (depends on 16)

The user device as recited by claim 16, wherein the one or more processors that are the multiple, different processors use pipeline processing to perform the super-resolution computations.

### Claim 19 (depends on 16)

The user device as recited by claim 16, wherein the instructions that direct the user device to perform the super-resolution computations include instructions that direct the user device to, as part of the super-resolution computations, perform Gaussian radial basis function kernel computations.

### Claim 20 (depends on 1)

The method as recited by claim 1, wherein the super-resolution computations include analyzing downscaling operations or detecting characteristic patterns.