# US 12,216,467 B2 · Self-location estimation device, autonomous mobile body, and self-location estimation method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,216,467 B2
- Document ID: US12216467B2
- Status: Granted
- Kind: B2
- Application number: 17/754,107
- Filed: 2020-09-23
- Published: 2022-09-15
- Granted: 2025-02-04
- Earliest claimed: 2019-10-01
- Assignees: Sony Semiconductor Solutions Corporation
- Applicants: Sony Semiconductor Solutions Corporation
- Inventors: Shin Yoshimura
- Web page: https://patentgrep.com/patent/US12216467B2

## CPC

- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G01C 21/30: Map- or contour-matching
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0272: {comprising means for registering the travel distance, e.g. revolutions of wheels (measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00)}
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G05D 1/243: Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals (using passive navigation aids external to the vehicle G05D1/244; using signals from positioning sensors located off-board the vehicle G05D1/249)
- G05D 1/2437: {Extracting relative motion information}
- G05D 1/246: using environment maps, e.g. simultaneous localisation and mapping [SLAM]
- G05D 1/2462: {using feature-based mapping}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- G06T 7/74: {involving reference images or patches}
- G01S 17/89: for mapping or imaging
- G05D 2105/32: {for amusement, e.g. toys}
- G05D 2107/40: Indoor domestic environment
- G05D 2109/12: with legs
- G05D 2111/10: Optical signals
- G06T 2207/20081: Training; Learning
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

A self-location estimation device includes a first self-location estimation unit, a second self-location estimation unit, and a first integration unit. The first self-location estimation unit estimates a current first self-location of an autonomous mobile body based on current image information acquired by an image sensor and environmental map information stored in an environmental map information storage unit. The second self-location estimation unit estimates a current second self-location of the autonomous mobile body based on the current image information and a learned parameter learned using the environmental map information. The first integration unit estimates a current self-location of the autonomous mobile body by integrating the first self-location and the second self-location.

## Claims

### Claim 1

A self-location estimation device, comprising:
a central processing unit (CPU) configured to:
receive current image information from one or more image sensors of an autonomous mobile body, wherein the autonomous mobile body is movable in a real space;
estimate a current first self-location of the autonomous mobile body based on the current image information and environmental map information, wherein
the environmental map information includes a plurality of pairs of past image information acquired in past by the one or more image sensors, feature data of the plurality of pairs of the past image information, and a past self-location of the autonomous mobile body at a time of the acquisition of each of the plurality of pairs of the past image information,
the plurality of pairs of the past image information, the feature data, and the past self-location of each of the plurality of pairs of the past image information are associated on a map in the environmental map information, and
the feature data corresponds to feature points of the plurality of pairs of the past image information;
estimate a current second self-location of the autonomous mobile body based on the current image information and a learned parameter, wherein the learned parameter is based on the environmental map information; and
integrate the current first self-location and the current second self-location to estimate a third current self-location of the autonomous mobile body.

### Claim 2 (depends on 1)

The self-location estimation device according to claim 1, wherein the CPU is further configured to:
generate a learning data set based on the environmental map information;
calculate, based on the learning data set, the learned parameter and specific estimation reliability of each of a plurality of regions on the map;
output each of the current second self-location and a self-location estimation reliability based on the estimation of the current second self-location and the learned parameter; and
estimate the third current self-location of the autonomous mobile body based on at least one of the specific estimation reliability of each of the plurality of regions or the self-location estimation reliability.

### Claim 3 (depends on 2)

The self-location estimation device according to claim 2, wherein the CPU is further configured to:
classify the plurality of pairs of the past image information into a plurality of groups;
set a reference self-location as a reference for each of the plurality of groups;
obtain, for each group of the plurality of groups, offset information that indicates an offset amount between the reference self-location and the past self-location for each pair of the plurality of pairs of the past image information;
set an input portion of the learning data set as the past image information of each pair of the plurality of pairs of the past image information; and
generate the learning data set in which an output portion of the learning data set is a list of each of the plurality of groups,
wherein each of the plurality of groups includes each of the plurality of pairs of the past image information, group information related to each of the plurality of groups, and the offset information.

### Claim 4 (depends on 2)

The self-location estimation device according to claim 2, wherein the CPU is further configured to:
determine whether meta-information of the plurality of pairs of the past image information is within an allowable range; and
exclude the feature data associated with a pair of the past image information from the plurality of pairs of the past image information in a case where the meta-information associated with the pair of the past information is not within the allowable range.

### Claim 5 (depends on 1)

The self-location estimation device according to claim 1, wherein the CPU is further configured to:
estimate a current fourth self-location based on odometry information of the autonomous mobile body;
calculate a weight coefficient that is multiplied by the current first self-location based on a self-location estimation reliability of the current first self-location;
integrate the current first self-location and the current fourth self-location based on the weight coefficient; and
estimate the third current self-location of the autonomous mobile body based on integration of an integration result of the current first self-location and the current fourth self-location with the current second self-location.

### Claim 6 (depends on 1)

The self-location estimation device according to claim 1, wherein the CPU is further configured to:
control a memory to store a plurality of different pieces of the environmental map information, wherein the plurality of different pieces has different relative locational relationships;
extract first feature data of the feature data from the past image information by a first method;
extract second feature data of the feature data from the past image information by a second method different from the first method; and
integrate the plurality of different pieces of the environmental map information based on the second feature data.

### Claim 7 (depends on 6)

The self-location estimation device according to claim 6, wherein the CPU is further configured to:
estimate the current first self-location based on the plurality of pairs of the past image information, the first feature data, the past self-location, and the current image information;
determine that the memory stores the plurality of different pieces of the environmental map information;
estimate similarity between the plurality of pairs of the past image information in the different pieces of the environmental map information, the first feature data, the second feature data, and the past self-location;
count a number of pairs, having similarity equal to or more than a threshold, in the plurality of different pieces of the environmental map information; and
integrate the plurality of different pieces of the environmental map information based on the past self-location of the number of pairs of which the similarity is equal to or more than the threshold, and the number of pairs is equal to or more than a determined value.

### Claim 8 (depends on 6)

The self-location estimation device according to claim 6, wherein
the CPU is further configured to execute a multi-object detection process on the past image information to detect the second feature data, and
wherein the second feature data includes at least one of a size, a location, a type, or a direction of an object.

### Claim 9 (depends on 4)

The autonomous mobile body according to claim 4, wherein
the meta-information corresponds to at least one of data capacity, image quality, image acquisition date, image acquisition time, identification information of the autonomous mobile body, photographing information, or noise information, and
the photographing information includes at least one of ISO, shutter speed, or F-number.

### Claim 10

A self-location estimation method comprising:
in a self-location estimation device:
receiving current image information from one or more image sensors of an autonomous mobile body, wherein the autonomous mobile body is movable in a real space;
estimating a current first self-location of the autonomous mobile body based on the current image information and environmental map information, wherein
the environmental map information includes a plurality of pairs of past image information acquired in past by the one or more image sensors, feature data of the plurality of pairs of the past image information, and a past self-location of the autonomous mobile body at a time of the acquisition of each of the plurality of pairs of the past image information,
the plurality of pairs of the past image information, the feature data, and the past self-location of each of the plurality of pairs of the past image information are associated on a map in the environmental map information, and
the feature data corresponds to feature points of the plurality of pairs of the past image information;
estimating a current second self-location of the autonomous mobile body based on the current image information and a learned parameter, wherein the learned parameter is based on the environmental map information; and
integrating the current first self-location and the current second self-location to estimate a third current self-location of the autonomous mobile body.

### Claim 11

A non-transitory computer-readable medium having stored thereon, computer-executable instructions, which when executed by a processor, cause the processor to execute operations, the operations comprising:
receiving current image information from one or more image sensors of an autonomous mobile body, wherein the autonomous mobile body is movable in a real space;
estimating a current first self-location of the autonomous mobile body based on the current image information and environmental map information, wherein
the environmental map information includes a plurality of pairs of past image information acquired in past by the one or more image sensors, feature data of the plurality of pairs of the past image information, and a past self-location of the autonomous mobile body at a time of the acquisition of each of the plurality of pairs of the past image information,
the plurality of pairs of the past image information, the feature data, and the past self-location of each of the plurality of pairs of the past image information are associated on a map in the environmental map information, and
the feature data corresponds to feature points of the plurality of pairs of the past image information;
estimating a current second self-location of the autonomous mobile body based on the current image information and a learned parameter, wherein the learned parameter is based on the environmental map information; and
integrating the current first self-location and the current second self-location to estimate a third current self-location of the autonomous mobile body.