# US 2022/0286604 A1 · Sliding window for image keypoint detection and descriptor generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0286604 A1
- Document ID: US20220286604A1
- Status: Published application
- Kind: A1
- Application number: 17/195,520
- Filed: 2021-03-08
- Published: 2022-09-08
- Earliest claimed: 2021-03-08
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: David R. Pope; Liran Fishel; Assaf Metuki; Muge Wang
- Web page: https://patentgrep.com/patent/US20220286604A1

## CPC

- H04N 5/76: Television signal recording
- G06T 1/60: Memory management
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 7/251: {involving models}
- H04N 1/2141: {in a multi-frame buffer}
- H04N 5/772: {the recording apparatus and the television camera being placed in the same enclosure}
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Embodiments relate to extracting features from images, such as by identifying keypoints and generating keypoint descriptors of the keypoints. An apparatus includes a pyramid image generator circuit, a keypoint descriptor generator circuit, and a pyramid image buffer. The pyramid image generator circuit generates an image pyramid from an input image. The keypoint descriptor generator circuit processes the pyramid images for keypoint descriptor generation. The pyramid image buffer stores different portions of the pyramid images generated by the pyramid image generator circuit at different times and provides the stored portions of the pyramid images to the keypoint descriptor generator circuit for keypoint descriptor generation. When first portions of the pyramid images are no longer needed for the keypoint descriptor generation, the first portions are removed from the pyramid image buffer to provide space for second portions of the pyramid images that are needed for the keypoint descriptor generation.

## Claims

### Claim 1

An apparatus, comprising:
a pyramid image generator circuit configured to generate an image pyramid from an input image, the image pyramid including pyramid images at different octaves and scales;
a keypoint descriptor generator circuit configured to process the pyramid images for keypoint descriptor generation of keypoints in the pyramid images; and
a pyramid image buffer connected to the pyramid image generator circuit and the keypoint descriptor generator circuit, the pyramid image buffer configured to store different portions of the pyramid images generated by the pyramid image generator circuit at different times, for sending to the keypoint descriptor generator circuit for the keypoint descriptor generation.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein, at any given time, the pyramid image buffer stores portions of full pyramid images of the image pyramid.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the pyramid image buffer has a storage size that is smaller than a data size of the image pyramid.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the pyramid image buffer is further configured to:
provide first portions of the pyramid images stored in memory locations of the pyramid image buffer to the keypoint descriptor generator circuit for the keypoint descriptor generation;
remove the first portions of the pyramid images from the memory locations responsive to the first portions being no longer needed for the keypoint descriptor generation by the keypoint descriptor generator circuit; and
store second portions of the pyramid images that are needed for the keypoint descriptor generation by the keypoint descriptor generator circuit in the memory locations that previously stored the first portions of the pyramid images.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the pyramid image buffer is further configured to:
provide the second portions of the pyramid images stored in the memory locations to the keypoint descriptor generator circuit for the keypoint descriptor generation; and
remove the second portions of the pyramid images from the memory locations when the second portions are no longer needed for the keypoint descriptor generation by the keypoint descriptor generator circuit.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein:
the pyramid image generator circuit, the keypoint descriptor generator circuit, and the pyramid image buffer are located on an integrated circuit (IC) chip; and
the pyramid image buffer is a static random-access memory (SRAM).

### Claim 7 (depends on 1)

The apparatus of claim 1, further comprising:
a response map (RM) generator circuit connected to the pyramid image generator circuit, the RM generator circuit configured to generate an RM image for each of pyramid images;
an inter octave keypoint selector circuit configured to process the RM images for keypoint determination of the keypoints in the pyramid images; and
an RM buffer connected to the RM generator circuit and the inter octave keypoint selector circuit, the RM buffer configured to store different portions of the RM images generated by the RM generator circuit at different times for sending to the inter octave keypoint selector circuit for the keypoint determination.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein the RM buffer is further configured to:
provide first portions of the RM images stored in memory locations of the RM buffer to the inter octave keypoint selector circuit for keypoint determination;
remove the first portions of the RM images from the memory locations responsive to the first portions being no longer needed for the keypoint determination by the inter octave keypoint selector circuit; and
store second portions of the RM images that are needed for the keypoint determination by the inter octave keypoint selector circuit in the memory locations that previously stored the first portions of the RM images.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the RM buffer is further configured to:
provide the second portions of the RM images stored in the memory locations to the inter octave keypoint selector circuit for keypoint determination; and
remove the second portions of the RM images from the memory locations responsive to the second portions being no longer needed for the keypoint determination by the inter octave keypoint selector circuit.

### Claim 10 (depends on 7)

The apparatus of claim 7, wherein, at any given time, the RM buffer does not store full RM images.

### Claim 11 (depends on 7)

The apparatus of claim 7, wherein the RM buffer has a storage size that is smaller than a data size of all RM images.

### Claim 12 (depends on 7)

The apparatus of claim 7, wherein:
the RM generator circuit, the inter octave keypoint selector circuit, and RM buffer are located on an integrated circuit (IC) chip; and
the RM buffer is a static random-access memory (SRAM).

### Claim 13 (depends on 7)

The apparatus of claim 7, further comprising an intra octave keypoint candidate selector circuit connected to the RM generator, the intra octave keypoint candidate selector circuit configured to determine keypoint candidates by applying intra octave non-maxima suppression (NMS) to the RM images, and wherein the inter octave keypoint selector circuit determines the keypoints from the keypoint candidates by applying inter octave NMS to the RM images.

### Claim 14 (depends on 7)

The apparatus of claim 7, wherein the inter octave keypoint selector circuit is configured to determine keypoints in multiple RM images simultaneously.

### Claim 15

A method, comprising:
generating, by a pyramid image generator circuit, an image pyramid from an input image, the image pyramid including pyramid images at different octaves and scales;
processing, by a keypoint descriptor generator circuit, the pyramid images for keypoint descriptor generation of keypoints in the pyramid images; and
storing, by a pyramid image buffer, different portions of the pyramid images generated by the pyramid image generator circuit at different times, for sending to the keypoint descriptor generator circuit for the keypoint descriptor generation.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
providing, by the pyramid image buffer, first portions of the pyramid images stored in memory locations of the pyramid image buffer to the keypoint descriptor generator circuit for the keypoint descriptor generation;
removing, by the pyramid image buffer, the first portions of the pyramid images from the memory locations responsive to the first portions being no longer needed for the keypoint descriptor generation by the keypoint descriptor generator circuit; and
storing, by the pyramid image buffer, second portions of the pyramid images that are needed for the keypoint descriptor generation by the keypoint descriptor generator circuit in the memory locations that previously stored the first portions of the pyramid images.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
generating, by a response map (RM) generator circuit, an RM image for each of pyramid images;
processing, by an inter octave keypoint selector circuit, the RM images for keypoint determination of the keypoints in the pyramid images; and
storing, by an RM buffer, different portions of the RM images generated by the RM generator circuit at different times for sending to the inter octave keypoint selector circuit for the keypoint determination.

### Claim 18

A system, comprising:
an image sensor configured to obtain an input image; and
an image signal processor connected to the image sensor, the image signal processor including:
a pyramid image generator circuit configured to generate an image pyramid from the input image, the image pyramid including pyramid images at different octaves and scales;
a keypoint descriptor generator circuit configured to process the pyramid images for keypoint descriptor generation of keypoints in the pyramid images; and
a pyramid image buffer connected to the pyramid image generator circuit and the keypoint descriptor generator circuit, the pyramid image buffer configured to store different portions of the pyramid images generated by the pyramid image generator circuit at different times for sending to the keypoint descriptor generator circuit for the keypoint descriptor generation.

### Claim 19 (depends on 18)

The system of claim 18, wherein the pyramid image buffer is further configured to:
provide first portions of the pyramid images stored in memory locations of the pyramid image buffer to the keypoint descriptor generator circuit for the keypoint descriptor generation;
remove the first portions of the pyramid images from the memory locations responsive to the first portions being no longer needed for the keypoint descriptor generation by the keypoint descriptor generator circuit; and
store second portions of the pyramid images that are needed for the keypoint descriptor generation by the keypoint descriptor generator circuit in the memory locations that previously stored the first portions of the pyramid images.

### Claim 20 (depends on 18)

The system of claim 18, wherein the image signal processor further includes:
a response map (RM) generator circuit connected to the pyramid image generator circuit, the RM generator circuit configured to generate an RM image for each of pyramid images;
an inter octave keypoint selector circuit configured to process the RM images for keypoint determination of the keypoints in the pyramid images; and
an RM buffer connected to the RM generator circuit and the inter octave keypoint selector circuit, the RM buffer configured to store different portions of the RM images generated by the RM generator circuit at different times for sending to the inter octave keypoint selector circuit for the keypoint determination.

### Claim 21

An apparatus, comprising:
a pyramid image generator circuit configured to generate an image pyramid from an input image, the image pyramid including pyramid images at different octaves and scales;
a keypoint descriptor generator circuit configured to process the pyramid images for keypoint descriptor generation of keypoints in the pyramid images; and
a pyramid image buffer connected to the pyramid image generator circuit and the keypoint descriptor generator circuit, the pyramid image buffer configured to:
provide first portions of the pyramid images stored in memory locations of the pyramid image buffer to the keypoint descriptor generator circuit for the keypoint descriptor generation;
remove the first portions of the pyramid images from the memory locations responsive to the first portions being no longer needed for the keypoint descriptor generation by the keypoint descriptor generator circuit; and
store second portions of the pyramid images that are needed for the keypoint descriptor generation by the keypoint descriptor generator circuit in the memory locations that previously stored the first portions of the pyramid images.
