# US 2025/0166352 A1 · Methods and systems of sensor fusion in cooperative perception systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166352 A1
- Document ID: US20250166352A1
- Status: Published application
- Kind: A1
- Application number: 18/838,458
- Filed: 2023-02-15
- Published: 2025-05-22
- Earliest claimed: 2022-02-15
- Assignees: Currus AI Inc.
- Applicants: Currus AI Inc.
- Inventors: Ehsan Emad Marvasti; Yaser Pourmohammadi Fallah; Hussein Alnuweiri
- Web page: https://patentgrep.com/patent/US20250166352A1

## CPC

- G06F 18/254: {of classification results, e.g. of results related to same input data}
- G06V 10/764: using classification, e.g. of video objects
- G06F 18/256: {of results relating to different input data, e.g. multimodal recognition}
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 7/50: Depth or shape recovery
- G06T 7/62: of area, perimeter, diameter or volume
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space
- G06V 10/806: {of extracted features}
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 2207/20081: Training; Learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Cooperative perception systems comprise a plurality of imaging sensors that are connected to provide output images to one of one or more machine learning (ML) systems, each ML system is trained to process the output images to yield variational hypotheses. Each of the variational hypotheses comprises one or more objects and, for each of the objects, values for each of a plurality of regressed parameters and variation data indicating uncertainty of the value for each of the plurality of regressed parameters. A processor receives and fuses the hypotheses using the variation data to yield a refined hypothesis. The refined hypothesis may provide an input to a control system for a vehicle, robot or other apparatus.

## Claims

### Claim 1

A cooperative perception system comprising:
a plurality of imaging sensors each of the imaging sensors connected to provide output images to one of one or more machine learning (ML) systems, the one or more ML systems trained to process the output images to yield hypotheses, each of the hypotheses comprising: one or more objects and, for each of the objects, values for each of a plurality of regressed parameters and variation data indicating uncertainty of the value for each of the plurality of regressed parameters; and
a processor connected to receive the hypotheses produced by the ML systems and to fuse the hypotheses using the variation data to yield a fused hypothesis.

### Claim at position 2

2-3. (canceled)

### Claim 4 (depends on 1)

The cooperative perception system according to claim 1 wherein each of the one or more ML systems is configured to output a precision matrix or covariance matrix that includes the variation data.

### Claim 5

(canceled)

### Claim 6 (depends on 1)

The cooperative perception system according to claim 1 wherein the ML systems are configured to classify the one or more objects into each of a plurality of classes and to output the values for each of a plurality of regressed parameters and variation data for each of the plurality of classes for each of the one or more objects.

### Claim 7 (depends on 1)

The cooperative perception system according to claim 1 wherein the variation data comprises an independent-component precision matrix and an associated rotation angle.

### Claim 8 (depends on 7)

The cooperative perception system according to claim 7 wherein the processor is configured to apply a rotation transformation based on the rotation angle to the independent-component precision matrix to yield a precision matrix in which off-diagonal terms indicate strengths and signs of correlations among the regressed parameter values.

### Claim at position 8

9-11. (canceled)

### Claim 12 (depends on 1)

The cooperative perception system according to claim 1 wherein, the variation data comprises a multivariate probability distribution and, in fusing the hypotheses, the processor is configured to compute products of the multivariate probability distributions of the hypotheses.

### Claim 13 (depends on 1)

The cooperative perception system according to claim 1 wherein the one or more ML system is trained to, for each of the objects, output a likelihood that the object belongs to each of a plurality of classes.

### Claim 14 (depends on 1)

The cooperative perception system according to claim 1 wherein the output images include a first set of one or more of the output images that are 2D images and a second set of the output images that are volumetric images.

### Claim 15 (depends on 14)

The cooperative perception system according to claim 14 wherein the ML systems connected to receive the first set of the output images comprise a depth channel and the regressed parameters include a depth estimate output by the depth channel.

### Claim 16

(canceled)

### Claim 17 (depends on 1)

The cooperative perception system according to claim 1 wherein the regressed parameters for the one or more objects comprise localization parameters that estimate a position of the object and one or more object size parameters that estimate a size of the object.

### Claim 18

(canceled)

### Claim 19 (depends on 1)

The cooperative perception system according to claim 1 wherein the regressed parameters of the one or more objects comprise one or more object size parameters that estimate size of the object in two or more dimensions.

### Claim 20 (depends on 1)

The cooperative perception system according to claim 1 wherein the processor is configured to filter the hypotheses to remove any of the hypotheses that have a confidence value below a confidence threshold before fusing the hypotheses.

### Claim 21

(canceled)

### Claim 22 (depends on 1)

The cooperative perception system according to claim 1 wherein the processor is configured to cluster the hypotheses, the clustering comprising:
calculate an entropy for each of the hypotheses;
select a hypothesis for which the entropy is lowest;
compute a divergence value between the selected hypothesis and the remaining hypotheses;
selecting for fusion the selected hypothesis and those of the remaining hypotheses for which the divergence value is lower than a divergence threshold.

### Claim at position 20

23-24. (canceled)

### Claim 25 (depends on 1)

The cooperative perception system according to claim 1 wherein a first variational hypothesis and a second variational hypothesis are derived from two-dimensional sensors and wherein the processor is configured to fuse the first variational hypothesis and the second variational hypothesis by:
projecting two or more 2D variational hypotheses into common 3D world coordinates;
identifying a point of closest approach;
estimating a piecewise conical approximation of each of the 2D variational hypotheses at a depth of the point of closest approach; and
fusing the piecewise conical approximation of the 2D variational hypotheses.

### Claim 26

A cooperative perception system comprising:
a plurality of imaging sensors each of the imaging sensors connected to provide output images to one of one or more first machine learning (ML) systems, the one or more ML systems comprising a plurality of layers and trained to process the output images to yield hypotheses, each of the hypotheses comprising: one or more objects and, for each of the objects, values for each of a plurality of regressed parameters and variation data indicating uncertainty of the value for each of the plurality of regressed parameters; and
one or more processors connected to:
receive feature maps from intermediate layers of the ML systems, the feature maps comprising partially-processed image data of the plurality of imaging sensors; and
fuse the feature maps to yield a fused feature map; and
process the fused feature map to yield a refined hypothesis, the refined hypotheses comprising: one or more objects and, for each of the objects, values for each of a plurality of regressed parameters.

### Claim 27 (depends on 26)

The cooperative perception system according to claim 26 wherein the feature maps each comprise a plurality of feature kernels, each of the feature kernels associated with a location and comprising a plurality of channels, each of the channels comprising a value.

### Claim 28

(canceled)

### Claim 29

The cooperative perception system according to 26 wherein the one or more processors comprises a second ML system configured to receive the fused feature map as input and to output the refined hypothesis.

### Claim 30 (depends on 26, 27, 28, 29)

The cooperative perception system according to any of claims 26 to 29 wherein the fused feature map comprises one or more feature tensors, the one or more processors are configured to populate one or more feature tensors with values from one or more sets of fused feature maps and the second ML system is configured to receive the one or more feature tensors as inputs and to output the refined hypothesis.

### Claim 31 (depends on 26)

The cooperative perception system according to claim 26 wherein the sensors comprise a set of first sensors having a first modality and a set of second sensors having a second modality, wherein the first modality is a 2D imaging modality and the second modality is a 3D imaging modality, and the one or more processors are configured to:
fuse a first set of feature maps corresponding to the first sensors,
fuse a second set of feature maps corresponding to the second sensors; and
combine the fused first and second sets of feature maps to yield the fused feature map.

### Claim at position 28

32-37. (canceled)

### Claim 38 (depends on 26)

The cooperative perception system according to claim 26 wherein the feature maps comprise categorical multivariate distributions.

### Claim at position 30

39-40. (canceled)

### Claim 41 (depends on 26)

The cooperative perception system according to claim 26 wherein the one or more processors are configured to cluster the feature maps, the clustering comprising:
calculating an entropy for each of the feature maps;
selecting a feature map for which the entropy is lowest;
computing a divergence value between the selected feature map and the remaining feature map; and
selecting for fusion the selected feature map and those of the remaining feature maps for which the divergence value is lower than a divergence threshold.

### Claim at position 32

42-81. (canceled)