# US 2025/0094535 A1 · Spatio-temporal cooperative learning for multi-sensor fusion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094535 A1
- Document ID: US20250094535A1
- Status: Published application
- Kind: A1
- Application number: 18/469,424
- Filed: 2023-09-18
- Published: 2025-03-20
- Earliest claimed: 2023-09-18
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Shivansh RAO; Sweta PRIYADARSHI; Varun RAVI KUMAR; Senthil Kumar YOGAMANI; Arunkumar NEHRUR RAVI; Vasudev BHASKARAN
- Web page: https://patentgrep.com/patent/US20250094535A1

## CPC

- G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F 18/251: {of input or preprocessed data}
- G06F 18/253: {of extracted features}
- G06V 10/806: {of extracted features}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- H04W 4/38: for collecting sensor information
- H04W 4/46: for vehicle-to-vehicle communication [V2V]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

According to aspects described herein, a device can extract first features from frames of first sensor data and second features from frames of second sensor data (captured after the first sensor data). The device can obtain first weighted features based on the first features and second weighted features based on the second features. The device can aggregate the first weighted features to determine a first feature vector and the second weighted features to determine a second feature vector. The device can obtain a first transformed feature vector (based on transforming the first feature vector into a coordinate space) and a second transformed feature vector (based on transforming the second feature vector into the coordinate space). The device can aggregate first transformed weighted features (based on the first transformed feature vector) and second transformed weighted features (based on the second transformed feature vector) to determine a fused feature vector.

## Claims

### Claim 1

A first device for processing data, the first device comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
extract first features from first frames of first sensor data;
extract second features from second frames of second sensor data, the second sensor data being captured after the first sensor data;
determine first weights associated with the first features to obtain first weighted features;
determine second weights associated with the second features to obtain second weighted features;
aggregate the first weighted features to determine a first feature vector;
aggregate the second weighted features to determine a second feature vector;
transform the first feature vector into a coordinate space to obtain a first transformed feature vector;
transform the second feature vector into the coordinate space to obtain a second transformed feature vector;
determine first transform weights associated with the first transformed feature vector to obtain first transformed weighted features;
determine second transform weights associated with the second transformed feature vector to obtain second transformed weighted features; and
aggregate the first transformed weighted features and the second transformed weighted features together to determine a fused feature vector.

### Claim 2 (depends on 1)

The first device of claim 1, wherein the at least one processor is configured to sense, using one or more sensors of the first device, an environment of the first device to obtain the first sensor data comprising the first frames.

### Claim 3 (depends on 2)

The first device of claim 2, wherein the one or more sensors comprises at least one of a camera, a radar sensor, or a light detection and ranging (LIDAR) sensor.

### Claim 4 (depends on 1)

The first device of claim 1, wherein the first device is a vehicle.

### Claim 5 (depends on 1)

The first device of claim 1, wherein the at least one processor is configured to receive, from one or more second devices, the second sensor data associated with an environment of the first device, wherein the second sensor data comprises the second frames.

### Claim 6 (depends on 5)

The first device of claim 5, wherein each second device of the one or more second devices is a vehicle.

### Claim 7 (depends on 1)

The first device of claim 1, wherein the first sensor data and the second sensor data each comprise at least one of camera sensor data, radar sensor data, or light detection and ranging (LIDAR) sensor data.

### Claim 8 (depends on 1)

The first device of claim 1, wherein the coordinate space is associated with the first device.

### Claim 9 (depends on 1)

The first device of claim 1, wherein the at least one processor is configured to:
determine the first weights based on performing a linear mapping and a normalization function on the first features; and
determine the second weights based on performing the linear mapping and the normalization function on the second features.

### Claim 10 (depends on 1)

The first device of claim 1, wherein the at least one processor is configured to transform the second feature vector into the coordinate space based on extracting features from wireless signals received from one or more second devices.

### Claim 11

A method for processing data by a first device, the method comprising:
extracting first features from first frames of first sensor data;
extracting second features from second frames of second sensor data, the second sensor data being captured after the first sensor data;
determining first weights associated with the first features to obtain first weighted features;
determining second weights associated with the second features to obtain second weighted features;
aggregating the first weighted features to determine a first feature vector;
aggregating the second weighted features to determine a second feature vector;
transforming the first feature vector into a coordinate space to obtain a first transformed feature vector;
transforming the second feature vector into the coordinate space to obtain a second transformed feature vector;
determining first transform weights associated with the first transformed feature vector to obtain first transformed weighted features;
determining second transform weights associated with the second transformed feature vector to obtain second transformed weighted features; and
aggregating the first transformed weighted features and the second transformed weighted features together to determine a fused feature vector.

### Claim 12 (depends on 11)

The method of claim 11, further comprising sensing, by one or more sensors of the first device, an environment of the first device to obtain the first sensor data comprising the first frames.

### Claim 13 (depends on 12)

The method of claim 12, wherein the one or more sensors comprises at least one of a camera, a radar sensor, or a light detection and ranging (LIDAR) sensor.

### Claim 14 (depends on 11)

The method of claim 11, wherein the first device is a vehicle.

### Claim 15 (depends on 11)

The method of claim 11, further comprising receiving, by the first device from one or more second devices, the second sensor data associated with an environment of the first device, wherein the second sensor data comprises the second frames.

### Claim 16 (depends on 15)

The method of claim 15, wherein each second device of the one or more second devices is a vehicle.

### Claim 17 (depends on 11)

The method of claim 11, wherein the first sensor data and the second sensor data each comprise at least one of camera sensor data, radar sensor data, or light detection and ranging (LIDAR) sensor data.

### Claim 18 (depends on 11)

The method of claim 11, wherein the coordinate space is associated with the first device.

### Claim 19 (depends on 11)

The method of claim 11, wherein:
the first weights are determined based on performing a linear mapping and a normalization function on the first features; and
the second weights are determined based on performing the linear mapping and the normalization function on the second features.

### Claim 20 (depends on 11)

The method of claim 11, wherein the second feature vector is transformed into the coordinate space based on extracting features from wireless signals received from one or more second devices.

### Claim 21

A non-transitory computer-readable storage medium of a first device comprising instructions stored thereon which, when executed by at least one processor, causes the at least one processor to:
extract first features from first frames of first sensor data;
extract second features from second frames of second sensor data, the second sensor data being captured after the first sensor data;
determine first weights associated with the first features to obtain first weighted features;
determine second weights associated with the second features to obtain second weighted features;
aggregate the first weighted features to determine a first feature vector;
aggregate the second weighted features to determine a second feature vector;
transform the first feature vector into a coordinate space to obtain a first transformed feature vector;
transform the second feature vector into the coordinate space to obtain a second transformed feature vector;
determine first transform weights associated with the first transformed feature vector to obtain first transformed weighted features;
determine second transform weights associated with the second transformed feature vector to obtain second transformed weighted features; and
aggregate the first transformed weighted features and the second transformed weighted features together to determine a fused feature vector.

### Claim 22 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein the instructions which, when executed by the at least one processor, cause the at least one processor to sense, using one or more sensors of the first device, an environment of the first device to obtain the first sensor data comprising the first frames.

### Claim 23 (depends on 22)

The non-transitory computer-readable storage medium of claim 22, wherein the one or more sensors comprises at least one of a camera, a radar sensor, or a light detection and ranging (LIDAR) sensor.

### Claim 24 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein the first device is a vehicle.

### Claim 25 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein the instructions which, when executed by the at least one processor, cause the at least one processor to receive, from one or more second devices, the second sensor data associated with an environment of the first device, wherein the second sensor data comprises the second frames.

### Claim 26 (depends on 25)

The non-transitory computer-readable storage medium of claim 25, wherein each second device of the one or more second devices is a vehicle.

### Claim 27 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein the first sensor data and the second sensor data each comprise at least one of camera sensor data, radar sensor data, or light detection and ranging (LIDAR) sensor data.

### Claim 28 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein the coordinate space is associated with the first device.

### Claim 29 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein the instructions which, when executed by the at least one processor, cause the at least one processor to:
determine the first weights based on performing a linear mapping and a normalization function on the first features; and
determine the second weights based on performing the linear mapping and the normalization function on the second features.

### Claim 30 (depends on 21)

The non-transitory computer-readable storage medium of claim 21, wherein the second feature vector is transformed into the coordinate space based on extracting features from wireless signals received from one or more second devices.
