# US 8,280,196 B2 · Image retrieval apparatus, control method for the same, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,280,196 B2
- Document ID: US8280196B2
- Status: Granted
- Kind: B2
- Application number: 12/764,660
- Filed: 2010-04-21
- Published: 2010-11-18
- Granted: 2012-10-02
- Earliest claimed: 2009-05-12
- Assignees: Canon Kabushiki Kaisha
- Inventors: Koichi Magai; Hirotaka Shiiyama
- Web page: https://patentgrep.com/patent/US8280196B2

## CPC

- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 10/757: {Matching configurations of points or features}
- G06V 30/2504: {Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image retrieval apparatus configured so as to enable a global feature method and a local feature method to complement each other is provided. After obtaining a retrieval result candidate using the local feature method, the image retrieval apparatus further verifies global features already registered in a database, with regard to the retrieval result candidate image. A verification position of the global features is estimated using the local features.

## Claims

### Claim 1

An image retrieval apparatus, comprising:
an extraction unit configured to extract a local feature amount, which is a localized feature amount, from an input image;
a retrieval unit configured to calculate a local feature similarity, which is a similarity between the extracted local feature amount and a local feature amount of each image registered in an image feature database, and acquire a plurality of candidate images based on the calculated local feature similarity;
a parameter calculation unit configured to calculate a geometric transformation parameter between each candidate image and the input image using the extracted local feature amount;
a determination unit configured to determine a calculation region on the input image corresponding to a global feature amount calculation region calculated in advance with respect to each candidate image, using the geometric transformation parameter;
a feature amount calculation unit configured to calculate a global feature amount in the calculation region determined by the determination unit;
a similarity calculation unit configured to calculate a similarity between each candidate image and the input image by comparing the global feature amount of each candidate image and the global feature amount of the input image; and
an output unit configured to output each candidate image in order of similarity.

### Claim 2 (depends on 1)

The image retrieval apparatus according to claim 1, further comprising a control unit configured to control execution of processing performed by the parameter calculation unit, the determination unit, the feature amount calculation unit, and the similarity calculation unit based on the local feature similarity calculated by the retrieval unit.

### Claim 3 (depends on 2)

The image retrieval apparatus according to claim 2,
wherein if a maximum value of the local feature similarity calculated by the retrieval unit is greater than or equal to a predetermined threshold value, the control unit performs control such that processing by the parameter calculation unit, the determination unit, the feature amount calculation unit, and the similarity calculation unit is not executed.

### Claim 4 (depends on 2)

The image retrieval apparatus according to claim 2,
wherein if a ratio of a maximum value and a second greatest value of the local feature similarity calculated by the retrieval unit is greater than or equal to a predetermined threshold value, the control unit performs control such that processing by the parameter calculation unit, the determination unit, the feature amount calculation unit, and the similarity calculation unit is not executed.

### Claim 5 (depends on 1)

The image retrieval apparatus according to claim 1, further comprising:
an edge detection unit configured to detect an edge in the calculation region on the input image determined by the determination unit; and
a control unit configured to perform control, if the edge detection unit detects an edge in the calculation region on the input image, such that processing by the parameter calculation unit, the determination unit, the feature amount calculation unit, and the similarity calculation unit is not executed.

### Claim 6 (depends on 1)

The image retrieval apparatus according to claim 1, further comprising a generation unit configured to generate a conjoined calculation region obtained by conjoining a plurality of global feature amount calculation regions on the input image obtained by the determination unit,
wherein the feature amount calculation unit calculates a global feature amount in the conjoined calculation region as a conjoined global feature amount, and
the similarity calculation unit calculates, by comparing a conjoined global feature amount of each candidate image and the conjoined global feature amount of the input image, a similarity between each candidate image and the input image.

### Claim 7 (depends on 6)

The image retrieval apparatus according to claim 6, wherein the plurality of calculation regions adjoin each other.

### Claim 8 (depends on 1)

The image retrieval apparatus according to claim 1, further comprising an image processing unit configured to perform image processing on a plurality of global feature amount calculation regions on the input image obtained by the determination unit,
wherein the feature amount calculation unit calculates a global feature amount from the plurality of calculation regions after image processing, and
the similarity calculation unit calculates, by comparing each global feature amount of the input image and a global feature amount after the image processing, a similarity between each candidate image and the input image.

### Claim 9 (depends on 1)

A non-transitory computer-readable storage medium having stored therein a program for causing a computer to execute the steps included in a control method for the image retrieval apparatus according to claim 1.

### Claim 10 (depends on 1)

The image retrieval apparatus according to claim 1, wherein the global feature amount comprises a color feature amount.

### Claim 11

A control method for an image retrieval apparatus, comprising:
an extraction step of extracting a local feature amount, which is a localized feature amount, from an input image;
a retrieval step of calculating a local feature similarity, which is a similarity between the extracted local feature amount and a local feature amount of each image registered in an image feature database, and acquiring a plurality of candidate images based on the calculated local feature similarity;
a parameter calculation step of calculating a geometric transformation parameter between each candidate image and the input image using the extracted local feature amount;
a determination step of determining a calculation region on the input image corresponding to a global feature amount calculation region calculated in advance with respect to each candidate image, using the geometric transformation parameter;
a feature amount calculation step of calculating a global feature amount in the calculation region determined by the determination unit;
a similarity calculation step of calculating a similarity between each candidate image and the input image by comparing the global feature amount of each candidate image and the global feature amount of the input image; and
an output step of outputting each candidate image in order of similarity.

### Claim 12

An image retrieval apparatus, comprising:
a local feature extraction unit configured to extract a local feature amount, which is a localized feature amount, from an input image;
an acquisition unit configured to calculate a local feature similarity, which is a similarity between the extracted local feature amount, and a local feature amount of each image registered in an image feature database, and acquire a plurality of candidate images based on the calculated local feature similarity;
an additional feature extraction unit configured to extract an additional feature, which is an added feature, from each of the input image and the acquired plurality of candidate images;
a calculation unit configured to calculate an additional feature similarity, which is a similarity between the additional feature extracted from each of the plurality of candidate images, and the additional feature extracted from the input image; and
an output unit configured to calculate a general similarity using the local feature similarity and the additional feature similarity, and output a predetermined number of candidate images as a retrieval result in order of calculated general similarity.

### Claim 13 (depends on 12)

The image retrieval apparatus according to claim 12, further comprising a normalization unit configured to perform normalization such that the plurality of candidate images acquired by the acquisition unit and the input image are matched in orientation and scale, and output the plurality of normalized candidate images to the additional feature extraction unit.

### Claim 14 (depends on 13)

The image retrieval apparatus according to claim 13,
wherein the normalization unit calculates a geometric transformation parameter between the plurality of candidate images and the input image, and performs the normalization using the calculated geometric transformation parameter.

### Claim 15 (depends on 12)

The image retrieval apparatus according to claim 12,
wherein the additional feature extraction unit excludes a region that extends beyond the input image when the input image and the candidate image are aligned and superimposed, from a region where the additional feature is extracted.

### Claim 16 (depends on 12)

The image retrieval apparatus according to claim 12,
wherein the additional feature extraction unit includes a judgment unit configured to judge, based on a spectrum distribution of a region where the additional feature is extracted, a type of additional feature to be extracted from the region.

### Claim 17 (depends on 12)

The image retrieval apparatus according to claim 12,
wherein the additional feature extraction unit includes an edge detection unit configured to detect an edge in a region where the additional feature is extracted, and if an edge is included in the region, excludes the region from a region where the additional feature is extracted.

### Claim 18 (depends on 12)

The image retrieval apparatus according to claim 12,
wherein if the local feature similarity is greater than or equal to a predetermined threshold value, the additional feature extraction unit does not perform processing.

### Claim 19

A control method for an image retrieval apparatus, comprising:
a local feature extraction step of extracting a local feature amount, which is a localized feature amount, from an input image;
an acquisition step of calculating a local feature similarity, which is a similarity between the extracted local feature amount, and a local feature amount of each image registered in an image feature database, and acquiring a plurality of candidate images based on the calculated local feature similarity;
an additional feature extraction step of extracting an additional feature, which is an added feature, from each of the input image and the acquired plurality of candidate images;
a calculation step of calculating an additional feature similarity, which is a similarity between the additional feature extracted from each of the plurality of candidate images, and the additional feature extracted from the input image; and
an output step of calculating a general similarity using the local feature similarity and the additional feature similarity, and outputting a predetermined number of candidate images as a retrieval result in order of calculated general similarity.

### Claim 20 (depends on 19)

A non-transitory computer-readable storage medium having stored therein a program for causing a computer to execute the steps included in a control method for the image retrieval apparatus according to claim 19.
