# US 12,118,800 B2 · Multiple hypothesis-based fusion of sensor data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,800 B2
- Document ID: US12118800B2
- Status: Granted
- Kind: B2
- Application number: 17/643,808
- Filed: 2021-12-10
- Published: 2022-08-18
- Granted: 2024-10-15
- Earliest claimed: 2021-02-15
- Assignees: Aptiv Technologies AG
- Applicants: Aptiv Technologies AG
- Inventors: Nianxia Cao; Xiaohui Wang
- Web page: https://patentgrep.com/patent/US12118800B2

## CPC

- G01S 13/931: of land vehicles
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G01S 13/72: for two-dimensional [2D] tracking, e.g. combination of angle and range tracking, track-while-scan radar
- G01S 13/867: {Combination of radar systems with cameras}
- G01S 15/66: Sonar tracking systems
- G01S 15/86: Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
- G01S 15/931: of land vehicles
- G01S 17/66: Tracking systems using electromagnetic waves other than radio waves
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/931: of land vehicles
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G01S 2013/9323: {Alternative operation using light waves}
- G01S 2013/9324: {Alternative operation using ultrasonic waves}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10044: Radar image
- G06T 2207/10132: Ultrasound image
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

This document describes a multiple hypothesis-based data fusion tracker. Each hypothesis aligns to a different pseudo measurement type. The fusion tracker automatically determines, using a predefined error covariance associated with the radar, which pseudo measurement type has a greater chance of being accurate for a current situation. The fusion tracker may rely on either one of two combined radar and vision calculations, or the fusion tracker may ignore the vision-based pseudo measurements and instead, rely on radar pseudo measurements alone. By selecting between three different bounding boxes, a vision angle based box, a vision lateral position based box, or a radar only based box, the fusion tracker can balance accuracy and speed when drawing, repositioning, or resizing bounding boxes, even under congested traffic or other high volume situations.

## Claims

### Claim 1

A method comprising:
determining, by a sensor-fusion system, a plurality of first object-tracks according to first sensor data obtained from a first set of sensors;
determining, by the sensor-fusion system, a set of second object-tracks according to second sensor data obtained from a second set of sensors;
selecting, from a plurality of pseudo measurement types, a pseudo measurement type that has a greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types, the selecting based at least in part on a predefined error covariance associated with the first set of sensors;
using the pseudo measurement type to determine a bounding box represented by the plurality of first object-tracks and the set of second object-tracks;
outputting an indication of the bounding box as a match between one or more object-tracks from the plurality of first object-tracks with at least one object-track from the set of second object-tracks;
wherein selecting the pseudo measurement type that has a greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types comprises:
determining a first probability of a first pseudo measurement derived from a radar range indicated by the one or more object-tracks from the plurality of first object-tracks and further derived from a vision angle inferred by the at least one object-track from the set of second object-tracks;
determining a second probability of a second pseudo measurement derived from the radar range and further derived from a vision lateral position inferred by the at least one object-track from the set of second object-tracks;
determining a third probability of a third pseudo measurement derived from radar information indicated by the one or more object-tracks from the plurality of first object-tracks, the radar information including the radar range, the third pseudo measurement not further derived from the vision angle or the vision lateral position; and
selecting, based on the first, second, and third probability, the pseudo measurement type that has the greater chance of being accurate.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first set of sensors are different than the second set of sensors.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first set of sensors comprise one or more radar sensors, lidar sensors, or ultrasound sensors and the second set of sensors comprise one or more visual sensors including optical cameras or infrared cameras.

### Claim 4 (depends on 1)

The method of claim 1, wherein selecting the pseudo measurement type that has the greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types comprises selecting the first pseudo measurement derived from the radar range indicated by the one or more object-tracks from the plurality of first object-tracks and further derived from the vision angle inferred by the at least one object-track from the set of second object-tracks.

### Claim 5 (depends on 4)

The method of claim 4, wherein a reference point (x,y) to the bounding box is represented in an x-direction as a product between the radar range and a cosine of the vision angle, and is represented in a y-direction as a product between the radar range and a sine of the vision angle.

### Claim 6 (depends on 1)

The method of claim 1, wherein selecting the pseudo measurement type that has the greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types comprises selecting the second pseudo measurement derived from the radar range indicated by the one or more object-tracks from the plurality of first object-tracks and further derived from the vision lateral position inferred by the at least one object-track from the set of second object-tracks.

### Claim 7 (depends on 6)

The method of claim 6, wherein a reference point (x,y) to the bounding box is represented in an x-direction as a square root of a difference between a square of the radar range and a square of the vision lateral position, and is represented in a y-direction as a product the vision lateral position.

### Claim 8 (depends on 1)

The method of claim 1, wherein selecting the pseudo measurement type that has the greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types comprises selecting the third pseudo measurement derived from radar information including the radar range indicated by the one or more object-tracks from the plurality of first object-tracks and not derived from either the vision angle or the vision lateral position inferred by the at least one object-track from the set of second object-tracks.

### Claim 9 (depends on 8)

The method of claim 8, wherein a reference point (x,y) to the bounding box is represented in a y-direction as the radar range.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
determining the error covariance associated with the first set of sensors.

### Claim 11 (depends on 1)

The method of claim 1, wherein selecting the pseudo measurement type that has the greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types comprises selecting the pseudo measurement type based on a history of pseudo measurement types selected based on previous frames of the first and second sensor data.

### Claim 12

A system, the system comprising:
a processor configured to:
determine a plurality of first object-tracks according to first sensor data obtained from a first set of sensors;
determine a set of second object-tracks according to second sensor data obtained from a second set of sensors;
select, from a plurality of pseudo measurement types, a pseudo measurement type that has a greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types, the selecting based at least in part on a predefined error covariance associated with the first set of sensors;
use the pseudo measurement type to determine a bounding box represented by the plurality of first object-tracks and the set of second object-tracks;
output an indication of the bounding box as a match between one or more object-tracks from the plurality of first object-tracks with at least one object-track from the set of second object-tracks;
wherein the processor is further configured to select the pseudo measurement type that has a greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types by at least selecting a first pseudo measurement derived from a radar range indicated by the one or more object-tracks from the plurality of first object-tracks and further derived from a vision angle inferred by the at least one object-track from the set of second object-tracks; and
wherein a reference point (x,y) to the bounding box is represented in an x-direction as a product between the radar range and a cosine of the vision angle, and is represented in a y-direction as a product between the radar range and a sine of the vision angle.

### Claim 13 (depends on 12)

The system of claim 12, further comprising:
a vehicle, the vehicle comprising the processor.

### Claim 14 (depends on 12)

The system of claim 12, wherein the first set of sensors are different than the second set of sensors.

### Claim 15 (depends on 12)

The system of claim 12, wherein the first set of sensors comprise one or more radar sensors, lidar sensors, or ultrasound sensors and the second set of sensors comprise one or more visual sensors including optical cameras or infrared cameras.

### Claim 16 (depends on 12)

The system of claim 12, wherein the processor is further configured to select the pseudo measurement type that has the greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types by at least selecting the second pseudo measurement derived from the radar range indicated by the one or more object-tracks from the plurality of first object-tracks and further derived from the vision lateral position inferred by the at least one object-track from the set of second object-tracks.

### Claim 17

A non-transitory computer-readable storage medium comprising instructions that, when executed, configure at least one processor of a system to:
determine a plurality of first object-tracks according to first sensor data obtained from a first set of sensors;
determine a set of second object-tracks according to second sensor data obtained from a second set of sensors;
select, from a plurality of pseudo measurement types, a pseudo measurement type that has a greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types, the selecting based at least in part on a predefined error covariance associated with the first set of sensors;
use the pseudo measurement type to determine a bounding box represented by the plurality of first object-tracks and the set of second object-tracks; and
output an indication of the bounding box as a match between one or more object-tracks from the plurality of first object-tracks with at least one object-track from the set of second object-tracks;
wherein the processor is further configured to select the pseudo measurement type that has a greater chance of being accurate than each other pseudo measurement type from the plurality of pseudo measurement types by at least selecting a first pseudo measurement derived from a radar range indicated by the one or more object-tracks from the plurality of first object-tracks and further derived from a vision angle inferred by the at least one object-track from the set of second object-tracks; and
wherein a reference point (x,y) to the bounding box is represented in an x-direction as a product between the radar range and a cosine of the vision angle, and is represented in a y-direction as a product between the radar range and a sine of the vision angle.