# US 11,112,252 B2 · Sensor fusion for accurate localization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,112,252 B2
- Document ID: US11112252B2
- Status: Granted
- Kind: B2
- Application number: 16/276,483
- Filed: 2019-02-14
- Published: 2020-08-20
- Granted: 2021-09-07
- Earliest claimed: 2019-02-14
- Assignees: Hitachi, Ltd.
- Applicants: Hitachi, Ltd.
- Inventors: Akshay Gokhale; Atsushi Yokoyama
- Web page: https://patentgrep.com/patent/US11112252B2

## CPC

- G01C 21/32: Structuring or formatting of map data
- G01S 17/89: for mapping or imaging
- G01C 21/30: Map- or contour-matching
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 7/497: Means for monitoring or calibrating
- G01S 13/86: Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 19/45: by combining measurements of signals from the satellite radio beacon positioning system with a supplementary measurement
- G01S 19/485: {whereby the further system is an optical system or imaging system}
- G06F 16/29: Geographical information databases
- G09B 29/106: {using electronic means}
- G01S 5/16: using electromagnetic waves other than radio waves
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

To localize a vehicle in the given map precisely for autonomous driving, the vehicle needs to compare the sensor perceived environment with the map information accurately. To achieve this, the vehicle may use information received from multiple on-board sensors in combination of GPS and map data. Example implementations described herein are directed to maintaining accuracy of sensor measurements for autonomous vehicles even during sensor failure through a raw data fusion technique and a repopulation of data through a prediction technique based on historical information and statistical method if any of the sensors fail.

## Claims

### Claim 1

A method for an autonomous vehicle having a plurality of sensors, the method comprising:
for a failure occurring in at least one of the plurality of sensors:
obtaining first sensor data points from working ones of the plurality of sensors;
obtaining second sensor data points from a historical database;
conducting a prediction of missing sensor data points based on an error correction process applied in comparing the first sensor data points and the second sensor data points;
weighing the missing sensor data points and the first sensor data points; and
applying the weighted missing sensor points and the first sensor data points to localization for autonomous vehicle functions of the autonomous vehicle.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
for a non-failure of the plurality of sensors:
transmitting sensor data points of the plurality of sensors for an associated timestamp to the historical database;
weighing the sensor data points; and
applying the weighted sensor points to the localization for the autonomous vehicle functions of the autonomous vehicle.

### Claim 3 (depends on 1)

The method of claim 1, wherein the historical database is managed in an apparatus external to the autonomous vehicle, the historical database comprises historical data of camera sensor data points and LIDAR sensor data points for a plurality of locations recorded from sensor data of a plurality of autonomous vehicles.

### Claim 4 (depends on 1)

The method of claim 1, wherein the conducting the prediction of the missing sensor data points comprises:
translating the received second sensor data points from the historical database according to local coordinates with respect to an orientation and location of the autonomous vehicle;
generating predictions of the missing data sensor points based on landmark objects associated with the translated second sensor data points within an error covariance; and
providing the predictions of the missing sensor data points having a lowest error covariance as the missing sensor data points.

### Claim 5 (depends on 1)

The method of claim 1, wherein the weighing of the missing sensor data points and the first sensor data points are based on a map matching process.

### Claim 6 (depends on 5)

The method of claim 5, wherein the map matching process comprises:
matching the missing sensor data points and the first data sensor points to associated points corresponding to landmarks in the historical database;
for matched points corresponding to camera and LIDAR ones of the points corresponding to the landmarks in the historical database, assign a first weight;
for matched points corresponding to only camera ones of the points corresponding to the landmarks in the historical database, assign a second weight that is less than the first weight;
for matched points corresponding to only LIDAR ones of the points corresponding to the landmarks in the historical database, assign a third weight that is less than the second weight.

### Claim 7

A non-transitory computer readable medium, storing instructions for an autonomous vehicle having a plurality of sensors, the instructions comprising:
for a failure occurring in at least one of the plurality of sensors:
obtaining first sensor data points from working ones of the plurality of sensors;
obtaining second sensor data points from a historical database;
conducting a prediction of missing sensor data points based on an error correction process applied in comparing the first sensor data points and the second sensor data points;
weighing the missing sensor data points and the first sensor data points; and
applying the weighted missing sensor points and the first sensor data points to localization for autonomous vehicle functions of the autonomous vehicle.

### Claim 8 (depends on 7)

The non-transitory computer readable medium of claim 7, the instructions further comprising:
for a non-failure of the plurality of sensors:
transmitting sensor data points of the plurality of sensors for an associated timestamp to the historical database;
weighing the sensor data points; and
applying the weighted sensor points to the localization for the autonomous vehicle functions of the autonomous vehicle.

### Claim 9 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the historical database is managed in an apparatus external to the autonomous vehicle, the historical database comprises historical data of camera sensor data points and LIDAR sensor data points for a plurality of locations recorded from sensor data of a plurality of autonomous vehicles.

### Claim 10 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the conducting the prediction of the missing sensor data points comprises:
translating the received second sensor data points from the historical database according to local coordinates with respect to an orientation and location of the autonomous vehicle;
generating predictions of the missing data sensor points based on landmark objects associated with the translated second sensor data points within an error covariance; and
providing the predictions of the missing sensor data points having a lowest error covariance as the missing sensor data points.

### Claim 11 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the weighing of the missing sensor data points and the first sensor data points are based on a map matching process.

### Claim 12 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the map matching process comprises:
matching the missing sensor data points and the first data sensor points to associated points corresponding to landmarks in the historical database;
for matched points corresponding to camera and LIDAR ones of the points corresponding to the landmarks in the historical database, assign a first weight;
for matched points corresponding to only camera ones of the points corresponding to the landmarks in the historical database, assign a second weight that is less than the first weight;
for matched points corresponding to only LIDAR ones of the points corresponding to the landmarks in the historical database, assign a third weight that is less than the second weight.

### Claim 13

An apparatus configured to manage an autonomous vehicle having a plurality of sensors, the apparatus comprising:
a processor, configured to:
for a failure occurring in at least one of the plurality of sensors:
obtain first sensor data points from working ones of the plurality of sensors;
obtain second sensor data points from a historical database;
conduct a prediction of missing sensor data points based on an error correction process applied in comparing the first sensor data points and the second sensor data points;
weigh the missing sensor data points and the first sensor data points; and
apply the weighted missing sensor points and the first sensor data points to localization for autonomous vehicle functions of the autonomous vehicle.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein for a non-failure of the plurality of sensors, the processor is configured to:
transmit sensor data points of the plurality of sensors for an associated timestamp to the historical database;
weigh the sensor data points; and
apply the weighted sensor points to the localization for the autonomous vehicle functions of the autonomous vehicle.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein the historical database is managed in another apparatus external to the autonomous vehicle, the historical database comprises historical data of camera sensor data points and LIDAR sensor data points for a plurality of locations recorded from sensor data of a plurality of autonomous vehicles.

### Claim 16 (depends on 13)

The apparatus of claim 13, wherein the processor is configured to conduct the prediction of the missing sensor data points by:
translating the received second sensor data points from the historical database according to local coordinates with respect to an orientation and location of the autonomous vehicle;
generating predictions of the missing data sensor points based on landmark objects associated with the translated second sensor data points within an error covariance; and
providing the predictions of the missing sensor data points having a lowest error covariance as the missing sensor data points.

### Claim 17 (depends on 13)

The apparatus of claim 13, wherein the weighing of the missing sensor data points and the first sensor data points are based on a map matching process.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein the processor is configured to execute the map matching process by:
matching the missing sensor data points and the first data sensor points to associated points corresponding to landmarks in the historical database;
for matched points corresponding to camera and LIDAR ones of the points corresponding to the landmarks in the historical database, assign a first weight;
for matched points corresponding to only camera ones of the points corresponding to the landmarks in the historical database, assign a second weight that is less than the first weight;
for matched points corresponding to only LIDAR ones of the points corresponding to the landmarks in the historical database, assign a third weight that is less than the second weight.