# US 2016/0239518 A1 · Method, a system, an apparatus and a computer program product for image-based retrieval

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0239518 A1
- Document ID: US20160239518A1
- Status: Published application
- Kind: A1
- Application number: 15/018,998
- Filed: 2016-02-09
- Published: 2016-08-18
- Earliest claimed: 2015-02-12
- Assignees: Nokia Technologies Oy
- Applicants: Nokia Technologies Oy
- Inventors: Kimmo Roimela; Lixin Fan
- Web page: https://patentgrep.com/patent/US20160239518A1

## CPC

- G06F 16/583: using metadata automatically derived from the content
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The invention relates to a method and a system for image-based retrieval. The method comprises receiving a query image; processing the query image to obtain visual features; determining a feature from the obtained visual features; determining a viewing direction for the query image from a pre-determined set of viewing directions; generating a direction-based hash based on the determined feature in the query image and the viewing direction; identifying sub-volumes of a point cloud database that match the direction-based hash; and matching the identified sub-volumes against all the query image features to find a match.

## Claims

### Claim 1

A method, comprising:
receiving a query image;
processing the query image to obtain visual features;
determining a feature from the visual features obtained by processing the query image;
determining a viewing direction for the query image from a pre-determined set of viewing directions;
generating a direction-based hash based on the feature in the query image and the viewing direction;
identifying sub-volumes of a point cloud database that match the direction-based hash; and
matching the sub-volumes against the query image features to find a match.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the viewing direction is predetermined viewing direction closest to an estimated viewing direction of the query image.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the feature is a strongest feature of the visual features.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising determining a location data relating to the query image based on the match, and outputting the location data.

### Claim 5 (depends on 1)

The method according to claim 1, wherein if a match is not found, the method further comprises setting a neighboring viewing direction of the viewing direction to be the viewing direction.

### Claim 6 (depends on 2)

The method according to claim 2, wherein the viewing direction of the query image is estimated based on sensor data.

### Claim 7 (depends on 1)

The method according to claim 1, wherein generating the direction-based hash comprises:
determining a quality metric for each feature within the viewing direction;
determining a strongest feature in the viewing direction by means of determined quality metrics;
generating a hash for the strongest feature; and
combining the hash with said the direction to generate the direction-based hash.

### Claim 8

An apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
receive a query image;
process the query image to obtain visual features;
determine a feature from the visual features obtained by processing the query image;
determine a viewing direction for the query image from a pre-determined set of viewing directions;
generate a direction-based hash based on the feature in the query image and the viewing direction;
identify sub-volumes of a point cloud database that match the direction-based hash; and
match the sub-volumes against the query image features to find a match.

### Claim 9 (depends on 8)

The apparatus according to claim 8, wherein the viewing direction is a predetermined viewing direction closest to an estimated viewing direction of the query image.

### Claim 10 (depends on 8)

The apparatus according to claim 8, wherein the feature is a strongest feature of the visual features.

### Claim 11 (depends on 8)

The apparatus according to claim 8, wherein the apparatus is further caused to determine a location data relating to the query image based on a match, and to output the location data.

### Claim 12 (depends on 8)

The apparatus according to claim 8, wherein if a match is not found, the apparatus is further caused to set a neighboring viewing direction of the viewing direction to be the viewing direction.

### Claim 13 (depends on 9)

The apparatus according to claim 9, wherein the viewing direction of the query image is estimated based on sensor data.

### Claim 14 (depends on 8)

The apparatus according to claim 8, wherein, to generate the direction-based hash, the apparatus is further caused to:
determine a quality metric for each feature within the viewing direction;
determine a strongest feature in said viewing direction by means of determined quality metrics;
generate a hash for the strongest feature; and
combine the hash with the viewing direction to generate the direction-based hash.

### Claim 15

A computer program product embodied on a non-transitory computer readable medium, comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to:
receive a query image;
process the query image to obtain visual features;
determine a feature from the visual features obtained by processing the query image;
determine a viewing direction for the query image from a pre-determined set of viewing directions;
generate a direction-based hash based on the feature in the query image and the viewing direction;
identify sub-volumes of a point cloud database that match the direction-based hash; and
match the sub-volumes against the query image features to find a match.

### Claim 16 (depends on 15)

The computer program product according to claim 15, wherein the viewing direction is a predetermined viewing direction closest to an estimated viewing direction of the query image.

### Claim 17 (depends on 15)

The computer program product according to claim 15, wherein the feature is a strongest feature of the visual features.

### Claim 18 (depends on 15)

The computer program product according to claim 15, wherein the apparatus or the system is further caused to determine a location data relating to the query image based on a match, and to output the location data.

### Claim 19 (depends on 15)

The computer program product according to claim 15, wherein if a match is not found, the apparatus or the system is further caused to set a neighboring viewing direction of the viewing direction to be the viewing direction.

### Claim 20 (depends on 16)

The computer program product according to claim 16, wherein the viewing direction of the query image is estimated based on sensor data.
