# US 2023/0408261 A1 · State estimation device and state estimation method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0408261 A1
- Document ID: US20230408261A1
- Status: Published application
- Kind: A1
- Application number: 18/309,225
- Filed: 2023-04-28
- Published: 2023-12-21
- Earliest claimed: 2022-06-16
- Assignees: Denso Corporation; Toyota Jidosha Kabushiki Kaisha; MIRISE Technologies Corporation
- Applicants: Denso Corporation; Toyota Jidosha Kabushiki Kaisha; MIRISE Technologies Corporation
- Inventors: NOBUAKI MATSUDAIRA; HUI XU
- Web page: https://patentgrep.com/patent/US20230408261A1

## CPC

- G01C 21/1656: {with passive imaging devices, e.g. cameras}
- G01C 25/005: {initial alignment, calibration or starting-up of inertial devices}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/73: using feature-based methods
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06T 2207/30244: Camera pose
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

By a state estimation device or a state estimation method, image data is read, a feature point included in the image data is extracted, the feature point is tracked, a position, a velocity, or an attitude of a mobile object is calculated based on inertia data, a bias error of an inertial measurement unit is calculated, correction data is calculated by removing the bias error from the inertia data, and a state including at least one of the position, the velocity, or the attitude of the mobile object is estimated based on the correction data.

## Claims

### Claim 1

A state estimation device for estimating a state including at least one of a position, a velocity, or an attitude of a mobile object, the device comprising:
an input unit configured to read image data output by a capture unit configured to capture an image of a peripheral area of the mobile object and inertia data of the mobile object, the inertia data being output from an inertial measurement unit installed on the mobile object;
a preprocessing unit configured to
extract a feature point included in the image data,
track the feature point, and
calculate the position, the velocity, or the attitude of the mobile object based on the inertia data;
a calculation unit configured to calculate a bias error of the inertial measurement unit by performing bundle adjustment on the feature point of the image data, the position, the velocity, or the attitude of the mobile object based on the inertia data;
a correction unit configured to calculate correction data by removing the bias error from the inertia data; and
an estimation unit configured to estimate a state including at least one of the position, the velocity, or the attitude of the mobile object based on the correction data.

### Claim 2 (depends on 1)

The state estimation device according to claim 1, wherein
the estimation unit is configured to
estimate the velocity of the mobile object based on the correction data and a sensor output of a vehicle wheel sensor installed on the mobile object, and
estimate the position of the mobile object based on the estimated speed of the mobile object.

### Claim 3 (depends on 2)

The state estimation device according to claim 2, wherein
the inertial measurement unit includes
a gyro sensor that detects an angular velocity of the mobile object and
an acceleration sensor that detects an acceleration of the mobile object, and
the estimation unit is configured to
calculate a first attitude angle indicating an attitude angle of the mobile object based on a result obtained by correction of a sensor output of the gyro sensor, the correction being performed by the correction unit,
calculate a second attitude angle indicating an attitude of the mobile object based on a gravitational acceleration calculated using a result obtained by correction of a sensor output of the acceleration sensor by the correction unit and a sensor output of the wheel speed sensor of the mobile object,
pass the first attitude angle through a high-pass filter of a complementary filter and passes the second attitude angle through a low-pass filter of the complementary filter, and
after passing the first attitude angle and the second attitude angle, synthesize the first attitude angle and the second attitude angle to estimate the attitude of the mobile object.

### Claim 4 (depends on 1)

The state estimation device according to claim 1, further comprising
an abnormality determination unit configured to determine whether an abnormal situation where the capture unit is difficult to perform a predetermined function has occurred,
wherein
the correction unit
calculates the correction data by removing the bias error from the inertia data when the abnormal situation has not occurred, and
calculates the correction data by removing, instead of the bias error calculated by the calculation unit, a bias error estimation value stored in a memory in advance from the inertia data when the abnormal situation has occurred.

### Claim 5 (depends on 4)

The state estimation device according to claim 4, wherein
the abnormality determination unit is configured to determine whether the abnormal situation has occurred based on the image data.

### Claim 6 (depends on 4)

The state estimation device according to claim 4, wherein
the abnormality determination unit is configured to determine whether the abnormal situation has occurred based on a sensor output of a steering angle sensor installed on the mobile object and an attitude of the mobile object, the attitude being calculated by the calculation unit.

### Claim 7 (depends on 4)

The state estimation device according to claim 4, wherein
the abnormality determination unit is configured to determine whether the abnormal situation has occurred based on a sensor output of an vehicle wheel speed sensor installed on the mobile object and a speed of the mobile object, the speed being calculated by the calculation unit.

### Claim 8 (depends on 4)

The state estimation device according to claim 4, wherein
the abnormality determination unit is configured to determine whether the abnormal situation has occurred based on the position, the velocity, and the attitude calculated by the calculation unit and the position, the velocity, and the attitude estimated by the estimation unit.

### Claim 9 (depends on 4)

The state estimation device according to claim 4, wherein
the correction unit is configured to
correct the bias error estimation value according to a temperature of the inertial measurement unit, and
calculate the correction data by removing the corrected bias error estimation value from the inertia data.

### Claim 10

A state estimation method for estimating a state including at least one of a position, a velocity, or an attitude of a mobile object, the method comprising:
reading image data output by a capture unit configured to capture an image of a peripheral area of the mobile object and inertia data of the mobile object, the inertia data being output from an inertial measurement unit installed on the mobile object;
extracting a feature point included in the image data;
tracking the feature point;
calculating the position, the velocity, or the attitude of the mobile object based on the inertia data;
calculating a bias error of the inertial measurement unit by performing bundle adjustment on the feature point of the image data and the position, the velocity, and the attitude of the mobile object based on the inertia data;
calculating correction data by removing the bias error from the inertia data; and
estimating a state including at least one of the position, the velocity, or the attitude of the mobile object based on the correction data.

### Claim 11 (depends on 1)

The state estimation device according to claim 1, further comprising
a processor that serves as the input unit, the preprocessing unit, the calculation unit, the correction unit, and the estimation unit.

### Claim 12 (depends on 4)

The state estimation device according to claim 4, further comprising
a processor that serves as the abnormality determination unit.