# US 2021/0080259 A1 · System and method for enhancing non-inertial tracking system with inertial constraints

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0080259 A1
- Document ID: US20210080259A1
- Status: Published application
- Kind: A1
- Application number: 16/569,024
- Filed: 2019-09-12
- Published: 2021-03-18
- Earliest claimed: 2019-09-12
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Benzun Pious Wisely Babu; Mao Ye; Liu Ren
- Web page: https://patentgrep.com/patent/US20210080259A1

## CPC

- G01C 21/165: {combined with non-inertial navigation instruments}
- G01C 21/1652: {with ranging devices, e.g. LIDAR or RADAR}
- G01C 21/1656: {with passive imaging devices, e.g. cameras}
- G01S 17/66: Tracking systems using electromagnetic waves other than radio waves
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A system and method for generating a. tracking state for a device includes synchronizing measurement data from exteroceptive sensors and an inertial measurement unit (IMU). A processing unit is programmed to offset one of the measurement signals by a time offset that minimizes a total error between a change in rotation of the device predicted by the exteroceptive sensor data over a time interval defined by an exteroceptive sensor sampling rate and a change in rotation of the device predicted by the IMU sensor data over the time interval.

## Claims

### Claim 1

A method for tracking position and orientation of a device comprising:
by a controller,
predicting a first tracking state of the device using measurements from an exteroceptive sensor (ES) sampled at a first rate;
generating a second tracking state of the device using measurements from an inertial measurement unit (IMU) sampled at a second rate that is asynchronous to the first rate;
synchronizing the first and second tracking states to a common time domain by estimating a time offset that minimizes an error between changes over time between the first and second tracking states such that a first time series associated with the first tracking state and a second time series associated with the second tracking state are aligned in time;
aligning the first tracking state and the second tracking state to an inertial reference frame; and
generating an inertial-aided tracking state by minimizing a cost function that includes a first residual term derived from propagating a previous state estimate using the second tracking state shifted by the time offset and a second residual term derived from the first tracking state.

### Claim 2 (depends on 1)

The method of claim 1 further comprising predicting the first tracking state by searching for a state estimate that minimizes a measurement error with respect to a set of observations, wherein motion states specific to the ES are in a local coordinate frame.

### Claim 3 (depends on 2)

The method of claim 2 further comprising generating a transformation to align the local coordinate frame to an inertial reference frame using a gravity vector from the IMU.

### Claim 4 (depends on 1)

The method of claim 1 further comprising predicting a first change in orientation based on a difference between most recent predictions of the first tracking state over a time interval defined by the first rate and predicting a second change in orientation based on the second. tracking state over the time interval.

### Claim 5 (depends on 4)

The method of claim 4 further comprising predicting the second change in orientation by integrating samples of the second tracking state over the time interval.

### Claim 6 (depends on 5)

The method of claim 5 wherein the second rate is greater than the first rate, and a predetermined number, based on a ratio of the first rate to the second rate, of samples from the IMU are integrated to predict the second change.

### Claim 7 (depends on 1)

The method of claim 1 further comprising minimizing the error using an optimization algorithm.

### Claim 8

A position and orientation tracking system for a device comprising:
a controller, in communication with an exteroceptive sensor core that samples an exteroceptive sensor at a first rate and generates a first set of measurement data and an inertial measurement unit (IMU) that samples IMU sensors at a second rate asynchronous to the first rate and generates a second set of measurement data, programmed to track motion of the device according to a tracking state derived by synchronizing the first and second sets of measurement data to a common time domain, wherein the first and second sets of measurement data are synchronized by offsetting one of the sets by a time offset that minimizes a. total error between a change in orientation of the device predicted by the first set of measurement data over a time interval defined by the first rate and a change in orientation of the device predicted by the second set of measurement data over the time interval.

### Claim 9 (depends on 8)

The system of claim 8, wherein the exteroceptive sensor is a camera.

### Claim 10 (depends on 8)

The system of claim 8, wherein the tracking state includes one or more of a position of the device, a velocity of the device, and an orientation. of the device.

### Claim 11 (depends on 8)

The system of claim 8, wherein the controller is further programmed to update the time offset with an amount of time that minimizes a distance between motion states estimated by a model predicting device motion based on the second set of measurement data and a model predicting device motion. based on the first set of measurement data that is shifted by the time offset and the amount of time.

### Claim 12 (depends on 8)

The system of claim 8, wherein controller is further programmed to receive an activity type that is based on IMU data, and transfer the activity type to the exteroceptive sensor core.

### Claim 13 (depends on 8)

The system of claim 8, wherein the controller is programmed to receive an activity type that is based on IMU data, and change a model for estimating the tracking state based on the activity type.

### Claim 14 (depends on 13)

The system of claim 13, wherein the controller is further programmed to estimate the tracking state by minimizing a cost function that includes a difference between the tracking state estimated by propagating a previous tracking state derived from the second set of measurement data and the tracking state predicted by the model based on the activity type.

### Claim 15 (depends on 8)

The system of claim 8, wherein the controller is further programmed to perform an optimization method to minimize the total error.

### Claim 16

An apparatus for synchronizing measurement data from an inertial measurement unit (IMU) and an exteroceptive sensor (ES) without a common electrical measurement trigger signal therebetween comprising:
a controller, in communication with an exteroceptive sensor core that is configured to sample an exteroceptive sensor at a first rate and an inertial measurement unit (IMU) that is configured to sample IMU sensors at a second rate asynchronous to the exteroceptive sensor, programmed to predict a first change in orientation based on a difference between most recent ES orientation data over a time interval defined by the first rate, predict a second change in orientation based on IMU orientation data over the time interval, and offset measurement data from the ES by a time offset that minimizes an error between the first change and the second change such that a first time series associated with the first change and a second time series associated with the second change are aligned in time.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the controller is further programmed to update the time offset with an amount of time that minimizes a distance between tracking states estimated by a model predicting the states based on IMU data and a model predicting the states based on exteroceptive sensor data that is shifted by the time offset.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein the controller is further programmed to receive an activity type that is based on IMU data and transfer the activity type to the exteroceptive sensor core.

### Claim 19 (depends on 16)

The apparatus of claim 16, wherein the controller is further programmed to perform an integration of sampled IMU data over the time interval for predicting the second change in rotation.

### Claim 20 (depends on 16)

The apparatus of claim 16, wherein the controller is further programmed to estimate a tracking state of a device using the time offset to shift, in time, a signal used for generating the tracking state.
