# US 2021/0279908 A1 · Estimation device, movable body, estimation method, and computer program product

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279908 A1
- Document ID: US20210279908A1
- Status: Published application
- Kind: A1
- Application number: 17/000,605
- Filed: 2020-08-24
- Published: 2021-09-09
- Earliest claimed: 2020-03-03
- Assignees: Kabushiki Kaisha Toshiba
- Applicants: Kabushiki Kaisha Toshiba
- Inventors: Ryo Nakashima; Akihito Seki
- Web page: https://patentgrep.com/patent/US20210279908A1

## CPC

- G06F 16/532: Query formulation, e.g. graphical querying
- G06F 16/583: using metadata automatically derived from the content
- G06F 16/587: using geographical or spatial information, e.g. location
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 20/00: Machine learning
- G06T 7/74: {involving reference images or patches}
- G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06N 3/045: Combinations of networks
- G06T 2207/30244: Camera pose
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An estimation device according to an embodiment includes one or more hardware processors configured to function as a query-image-acquisition unit, a query-imaging-condition-acquisition unit, a reference-image-acquisition unit, a reference-imaging-condition-acquisition unit, a feature-amount calculation unit, and a self-position calculation unit. The query-imaging-condition-acquisition unit acquires an imaging condition of a query image. The reference-imaging-condition-acquisition unit acquires an imaging condition of a reference image. The feature-amount calculation unit calculates a query image's feature based on the reference imaging condition, and calculates a reference image's feature based on the query imaging condition. The self-position calculation unit associates the reference image with the query image based on the query feature and the reference feature, and uses a position/pose of a second imaging device capturing the reference image associated with the query image to calculate a self location indicating at least one of a position/pose of a first imaging device.

## Claims

### Claim 1

An estimation device comprising:
one or more hardware processors configured to function as:
a query image acquisition unit that acquires an image captured by a first imaging device as a query image;
a query imaging condition acquisition unit that acquires a query imaging condition indicating an imaging condition of the query image;
a reference image acquisition unit that acquires at least one reference image;
a reference imaging condition acquisition unit that acquires a reference imaging condition indicating the imaging condition of the reference image;
a feature calculation unit that calculates a query feature indicating a feature of the query image from the query image based on the reference imaging condition, and calculates a reference feature indicating a feature of the reference image from the reference image based on the query imaging condition; and
a self-position calculation unit that associates the reference image with the query image based on the query feature and the reference feature, and uses a position and a pose of a second imaging device capturing the reference image associated with the query image to calculate a self location indicating at least one of a position and a pose of the first imaging device.

### Claim 2 (depends on 1)

The estimation device according to claim 1, wherein
the self-position calculation unit calculates a first global similarity indicating a similarity between the query image and the reference image based on the query feature and the reference feature, associates the reference image whose first global similarity is greater than a first image comparison threshold with the query image, and calculates the self location using the position and pose of the second imaging device capturing the reference image associated with the query image.

### Claim 3 (depends on 2)

The estimation device according to claim 2, wherein the one or more hardware processors are configured to function as
a second global similarity calculation unit that calculates a second global similarity between the query image and the reference image defined based on a feature that is non-dependent on the query imaging condition and the reference imaging condition, wherein
the self-position calculation unit calculates the first global similarity targeting a reference image whose second global similarity is greater than a second image comparison threshold, associates a reference image whose first global similarity is greater than the first image comparison threshold with the query image, and calculates the self location using the position and the pose of the second imaging device capturing the reference image associated with the query image.

### Claim 4 (depends on 1)

The estimation device according to claim 1, wherein
the self-position calculation unit calculates a local similarity indicating a similarity between a pixel of the reference image and a pixel of the query image based on the query feature and the reference feature, associates the pixel of the reference image whose local similarity is greater than a pixel comparison threshold with the pixel of the query image, and uses a geometric constraint defined from the associated pixel to calculate the self location.

### Claim 5 (depends on 1)

The estimation device according to claim 1, wherein
the query imaging condition and the reference imaging condition include at least one of a latitude, a longitude, a date, a time, a precipitation amount, and a snowfall amount at a time of imaging.

### Claim 6 (depends on 1)

The estimation device according to claim 1, wherein the one or more hardware processors are configured to function as
a machine-learning model that estimates the query imaging condition and the reference imaging condition, wherein
the query imaging condition acquisition unit acquires the query imaging condition from the machine-learning model, and the reference imaging condition acquisition unit acquires the reference imaging condition from the machine-learning model.

### Claim 7 (depends on 6)

The estimation device according to claim 6, wherein
the machine-learning model is achieved by a first neural network, and estimates the query imaging condition and the reference imaging condition by way of outputting by an output layer of the first neural network or outputting by an intermediate layer of the first neural network.

### Claim 8 (depends on 1)

The estimation device according to claim 1, wherein
the feature calculation unit is achieved by a second neural network.

### Claim 9

A movable body comprising:
a first imaging device that captures a query image;
a query image acquisition unit that acquires the query image from the first imaging device;
a query imaging condition acquisition unit that acquires a query imaging condition indicating an imaging condition of the query image;
a reference image acquisition unit that acquires at least one reference image;
a reference imaging condition acquisition unit that acquires a reference imaging condition indicating the imaging condition of the reference image;
a feature calculation unit that calculates a query feature indicating a feature of the query image from the query image based on the reference imaging condition, and calculates a reference feature indicating a feature of the reference image from the reference image based on the query imaging condition;
a self-position calculation unit that associates the reference image with the query image based on the query feature and the reference feature, and uses a position and a pose of a second imaging device capturing the reference image associated with the query image to calculate a self location indicating at least one of a position and a pose of the first imaging device; and
a power control unit that controls a power unit based on the self location.

### Claim 10

An estimation method comprising:
acquiring, by an estimation device comprising one or more hardware processors, an image captured by a first imaging device as a query image;
acquiring, by the estimation device, a query imaging condition indicating an imaging condition of the query image;
acquiring, by the estimation device, at least one reference image;
acquiring, by the estimation device, a reference imaging condition indicating the imaging condition of the reference image;
calculating, by the estimation device, a query feature indicating a feature of the query image from the query image based on the reference imaging condition, and a reference feature indicating a feature of the reference image from the reference image based on the query imaging condition; and
associating, by the estimation device, the reference image with the query image based on the query feature and the reference feature, and using a position and a pose of a second imaging device capturing the reference image associated with the query image to calculate a self location indicating at least one of a position and a pose of the first imaging device.

### Claim 11

A computer program product including programmed instructions embodied in and stored on a non-transitory computer readable medium, wherein the instructions, when executed by a computer, cause the computer to function as:
a query image acquisition unit that acquires an image captured by a first imaging device as a query image;
a query imaging condition acquisition unit that acquires a query imaging condition indicating an imaging condition of the query image;
a reference image acquisition unit that acquires at least one reference image;
a reference imaging condition acquisition unit that acquires a reference imaging condition indicating the imaging condition of the reference image;
a feature calculation unit that calculates a query feature indicating a feature of the query image from the query image based on the reference imaging condition, and calculates a reference feature indicating a feature of the reference image from the reference image based on the query imaging condition; and
a self-position calculation unit that associates the reference image with the query image based on the query feature and the reference feature, and uses a position and a pose of a second imaging device capturing the reference image associated with the query image to calculate a self location indicating at least one of a position and a pose of the first imaging device.