# US 2024/0005518 A1 · Scale image resolution for motion estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005518 A1
- Document ID: US20240005518A1
- Status: Published application
- Kind: A1
- Application number: 17/853,422
- Filed: 2022-06-29
- Published: 2024-01-04
- Earliest claimed: 2022-06-29
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Nitin BANDWAR; Pia ZOBEL; Roee HARDOON; Sungwon LEE; Tauseef KAZI; Bing HAN
- Web page: https://patentgrep.com/patent/US20240005518A1

## CPC

- G06T 7/207: for motion estimation over a hierarchy of resolutions (multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53)
- G06T 7/248: {involving reference images or patches}
- G06T 7/215: Motion-based segmentation
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and techniques are provided for conserving resources when performing motion estimation. An example process can include determining, based on an input image and a reference image, motion vectors identifying motion between the input image and the reference image; determining whether the motion indicated by the motion vectors is below a first threshold; based on a determination that the motion indicated by the motion vectors is below the first threshold, refraining from determining a local motion between the input image and the reference image; determining a transform matrix based on the motion vectors and without using a local motion between the input image and the reference image; and adjusting the input image based on the transform matrix.

## Claims

### Claim 1

An apparatus for processing image data, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
determine, based on an input image and a reference image, motion vectors indicating motion between the input image and the reference image;
determine whether the motion indicated by the motion vectors is below a first threshold;
based on a determination that the motion indicated by the motion vectors is below the first threshold, refrain from determining a local motion between the input image and the reference image;
determine a transform matrix based on the motion vectors and without using the local motion between the input image and the reference image; and
adjust the input image based on the transform matrix.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein, to adjust the input image based on the transform matrix, the at least one processor is configured to stabilize the input image using the transform matrix.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein, to determine the transform matrix based on the motion vectors, the at least one processor is configured to warp the motion vectors using a global stabilization matrix.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is configured to:
determine, based on a second input image and a second reference image, additional motion vectors identifying motion between the second input image and the second reference image;
determine whether the motion indicated by the additional motion vectors is below or above the first threshold;
based on a determination that the motion indicated by the additional motion vectors is above the first threshold, generate a motion map indicating a local motion between the second input image and the second reference image;
determine an additional transform matrix based on the motion map; and
adjust the second input image based on the additional transform matrix.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein, to determine an additional transform matrix based on the motion map, the at least one processor is configured to warp the motion map using a global stabilization matrix.

### Claim 6 (depends on 4)

The apparatus of claim 4, wherein the at least one processor is configured to:
prior to generating the motion map, align the additional motion vectors based on a global motion associated with the apparatus, the additional motion vectors being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit; and
generate the motion map based on the aligned additional motion vectors.

### Claim 7 (depends on 4)

The apparatus of claim 4, wherein the at least one processor is configured to determine that the motion indicated by the additional motion vectors is above the first threshold and below a second threshold that is greater than the first threshold.

### Claim 8 (depends on 4)

The apparatus of claim 4, wherein the at least one processor is configured to:
prior to determining the additional motion vectors identifying motion between the second input image and the second reference image, downscale the second input image and the second reference image, wherein the additional motion vectors are determined based on the downscaled second input image and the downscaled second reference image.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is configured to:
determine, based on a second input image and a second reference image, additional motion vectors identifying motion between the second input image and the second reference image;
determine whether the motion indicated by the additional motion vectors is above the first threshold and below a second threshold that is greater than the first threshold; and
based on a determination that the motion indicated by the additional motion vectors is above the first threshold and below the second threshold:
downscale the second input image and the second reference image;
generate a motion map indicating a local motion between the downscaled second input image and the downscaled second reference image;
determine an additional transform matrix based on the motion map; and
adjust the second input image based on the additional transform matrix.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein, to determine an additional transform matrix based on the motion map, the at least one processor is configured to warp the motion map using a global stabilization matrix.

### Claim 11 (depends on 9)

The apparatus of claim 9, wherein the at least one processor is configured to:
prior to generating the motion map, align the additional motion vectors based on a global motion associated with the apparatus, the additional motion vectors being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein, to generate the motion map, the at least one processor is configured to generate the motion map based on the aligned additional motion vectors.

### Claim 13 (depends on 1)

The apparatus of claim 1, further comprising a camera configured to capture a video sequence comprising the input image and the reference image.

### Claim 14 (depends on 1)

The apparatus of claim 1, wherein the apparatus comprises a mobile device.

### Claim 15

A method of processing image data, the method comprising:
determining, by a computing device based on an input image and a reference image, motion vectors indicating motion between the input image and the reference image;
determining, by the computing device, whether the motion indicated by the motion vectors is below a first threshold;
based on a determination that the motion indicated by the motion vectors is below the first threshold, refraining from determining a local motion between the input image and the reference image;
determining, by the computing device, a transform matrix based on the motion vectors and without using the local motion between the input image and the reference image; and
adjusting, by the computing device, the input image based on the transform matrix.

### Claim 16 (depends on 15)

The method of claim 15, wherein adjusting the input image based on the transform matrix comprises stabilizing the input image using the transform matrix.

### Claim 17 (depends on 15)

The method of claim 15, wherein determining the transform matrix based on the motion vectors comprises warping the motion vectors using a global stabilization matrix.

### Claim 18 (depends on 15)

The method of claim 15, further comprising:
determining, based on a second input image and a second reference image, additional motion vectors identifying motion between the second input image and the second reference image;
determining whether the motion indicated by the additional motion vectors is below or above the first threshold;
based on a determination that the motion indicated by the additional motion vectors is above the first threshold, generating a motion map indicating a local motion between the second input image and the second reference image;
determining an additional transform matrix based on the motion map; and
adjusting the second input image based on the additional transform matrix.

### Claim 19 (depends on 18)

The method of claim 18, wherein determining an additional transform matrix based on the motion map comprises warping the motion map using a global stabilization matrix.

### Claim 20 (depends on 18)

The method of claim 18, further comprising:
prior to generating the motion map, aligning the additional motion vectors based on a global motion associated with the computing device, the additional motion vectors being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit; and
generating the motion map based on the aligned additional motion vectors.

### Claim 21 (depends on 18)

The method of claim 18, further comprising determining that the motion indicated by the additional motion vectors is above the first threshold and below a second threshold that is greater than the first threshold.

### Claim 22 (depends on 18)

The method of claim 18, further comprising:
prior to determining the additional motion vectors identifying motion between the second input image and the second reference image, downscaling the second input image and the second reference image, wherein the additional motion vectors are determined based on the downscaled second input image and the downscaled second reference image.

### Claim 23 (depends on 15)

The method of claim 15, further comprising:
determining, based on a second input image and a second reference image, additional motion vectors identifying motion between the second input image and the second reference image;
determining whether the motion indicated by the additional motion vectors is above the first threshold and below a second threshold that is greater than the first threshold; and
based on a determination that the motion indicated by the additional motion vectors is above the first threshold and below the second threshold:
downscaling the second input image and the second reference image;
generating a motion map indicating a local motion between the downscaled second input image and the downscaled second reference image;
determining an additional transform matrix based on the motion map; and
adjusting the second input image based on the additional transform matrix.

### Claim 24 (depends on 23)

The method of claim 23, wherein determining an additional transform matrix based on the motion map comprising warping the motion map using a global stabilization matrix.

### Claim 25 (depends on 23)

The method of claim 23, further comprising:
prior to generating the motion map, aligning the additional motion vectors based on a global motion associated with the computing device, the additional motion vectors being aligned based on motion information obtained from a motion sensor, the motion sensor comprising at least one of a gyroscope and an inertial measurement unit.

### Claim 26 (depends on 25)

The method of claim 25, wherein generating the motion map comprises generating the motion map based on the aligned additional motion vectors.

### Claim 27

A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:
determine, based on an input image and a reference image, motion vectors indicating motion between the input image and the reference image;
determine whether the motion indicated by the motion vectors is below a first threshold;
based on a determination that the motion indicated by the motion vectors is below the first threshold, refrain from determining a local motion between the input image and the reference image;
determine a transform matrix based on the motion vectors and without using the local motion between the input image and the reference image; and
adjust the input image based on the transform matrix.

### Claim 28 (depends on 27)

The non-transitory computer-readable medium of claim 27, wherein, to adjust the input image based on the transform matrix, the instructions, when executed by the one or more processors, cause the one or more processors to stabilize the input image using the transform matrix.

### Claim 29 (depends on 27)

The non-transitory computer-readable medium of claim 27, wherein, to determine the transform matrix based on the motion vectors, the instructions, when executed by the one or more processors, cause the one or more processors to warp the motion vectors using a global stabilization matrix.

### Claim 30 (depends on 27)

The non-transitory computer-readable medium of claim 27, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:
determine, based on a second input image and a second reference image, additional motion vectors identifying motion between the second input image and the second reference image;
determine whether the motion indicated by the additional motion vectors is below or above the first threshold;
based on a determination that the motion indicated by the additional motion vectors is above the first threshold, generate a motion map indicating a local motion between the second input image and the second reference image;
determine an additional transform matrix based on the motion map; and
adjust the second input image based on the additional transform matrix.
