# US 10,120,879 B2 · Scalable attribute-driven image retrieval and re-ranking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,120,879 B2
- Document ID: US10120879B2
- Status: Granted
- Kind: B2
- Application number: 14/555,477
- Filed: 2014-11-26
- Published: 2015-06-04
- Granted: 2018-11-06
- Earliest claimed: 2013-11-29
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Le An; Changjian Zou; Liyan Zhang; Bradley Scott Denney; Dariusz Dusberger
- Web page: https://patentgrep.com/patent/US10120879B2

## CPC

- G06F 16/583: using metadata automatically derived from the content
- G06F 18/2178: {based on feedback of a supervisor}
- G06F 18/24137: {Distances to cluster centroïds}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7784: {based on feedback from supervisors}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06F 18/2132: based on discrimination criteria, e.g. discriminant analysis
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Retrieval of images of objects from a large-scale database of object images, based on a query image. The database may, for example, contain images of objects such as faces, vehicles, people and luggage. Semantic attributes such as doors or windows in the case of vehicles are used as high level semantic cues to determine identities of objects in the images. Salient visual characteristics of the images are labeled with attribute information, and a transformation is learned so as to transform the labeled visual characteristics into a discrimination vector that discriminates between the labels. A similarity metric is learned using the discrimination vectors, such that different images depicting the same object are determined to be close while those having different objects are determined to be far apart. Candidates are retrieved based on a query image, and a re-ranking step may be applied to improve results. Validation experiments are described.

## Claims

### Claim 1

A method for retrieval of images from a large-scale database of images, comprising:
obtaining a user designation of a query image;
extracting features representative of the query image;
searching the large-scale database of images to obtain a candidate set of images from the large-scale database of images based at least in part on similarity of the features for the query image to features of the images in the large-scale database of images;
ranking the obtained candidate images;
projecting the query image into a subspace that captures semantic attributes; and
re-ranking the ranked candidate images based at least in part on similarity of a result of projecting the query image into the subspace that captures semantic attributes to a result of projecting the images in the large-scale database of images into the subspace that captures semantic attributes, wherein the similarity of the result of projecting the query image into the subspace that captures semantic attributes to the result of projecting the images in the large-scale database of images into the subspace that captures semantic attributes is measured by a distance metric that measures distances between attributes.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the projection into the subspace that captures semantic attributes, and the distance metric, are both learned in a training phase from the large-scale database of images labeled with semantic attributes.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the projection into the subspace that captures semantic attributes uses linear discriminant analysis (LDA) from part-based HOG (Histogram of Oriented Gradients) features with human attributes as LDA class labels.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising object detection and alignment of the query image using algorithms that are also applied to the images in the large-scale database of images during a training phase from the large-scale database of images.

### Claim 5 (depends on 1)

The method according to claim 1, wherein re-ranking is also based at least in part on similarity of the features for the query image to features of the candidate images, using a weighting factor that applies respective weights to similarity of the projection and to similarity of the features.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the weighting factor is learned in a training phase from the large-scale database of images labeled with semantic attributes.

### Claim 7

An apparatus for retrieval of images from a large-scale database of images, comprising:
an interface to a large-scale database of multiple images;
memory for storing computer-executable process steps; and
one or more processors for executing the computer-executable process steps stored in the memory;
wherein the computer-executable process steps include steps to:
obtain a user designation of a query image;
extract features representative of the query image;
search the large-scale database of images to obtain a candidate set of images from the large-scale database of images based at least in part on similarity of the features for the query image to features of the images in the large-scale database of images;
rank the obtained candidate images;
project the query image into a subspace that captures semantic attributes; and
re-rank the ranked candidate images based at least in part on similarity of a result of projecting the query image into the subspace that captures semantic attributes to a result of projecting the images in the large-scale database of images into the subspace that captures semantic attributes, wherein the similarity of the result of projecting the query image into the subspace that captures semantic attributes to the result of projecting the images in the large-scale database of images into the subspace that captures semantic attributes is measured by a distance metric that measures distances between attributes.

### Claim 8 (depends on 1)

A non-transitory computer-readable memory medium which retrievably stores computer-executable process steps for causing a computer to perform the method according to claim 1.