# US 2024/0200969 A1 · Machine learning based occupancy grid generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0200969 A1
- Document ID: US20240200969A1
- Status: Published application
- Kind: A1
- Application number: 18/067,798
- Filed: 2022-12-19
- Published: 2024-06-20
- Earliest claimed: 2022-12-19
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Volodimir SLOBODYANYUK; Radhika Dilip GOWAIKAR; Makesh Pravin JOHN WILSON; Shantanu Chaisson SANYAL; Avdhut JOSHI; Christopher BRUNNER; Behnaz REZAEI; Amin ANSARI
- Web page: https://patentgrep.com/patent/US20240200969A1

## CPC

- G01C 21/3807: {characterised by the type of data}
- G01S 13/865: {Combination of radar systems with lidar systems}
- G01C 21/3841: {Data obtained from two or more sources, e.g. probe vehicles}
- G01S 7/417: {involving the use of neural networks}
- G01S 13/867: {Combination of radar systems with cameras}
- G01S 13/931: of land vehicles
- G05B 13/027: {using neural networks only}
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G08G 1/0104: {Measuring and analyzing of parameters relative to traffic conditions}
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

In some aspects, a device may receive sensor data associated with a vehicle and a set of frames. The device may aggregate, using a first pose, the sensor data associated with the set of frames to generate an aggregated frame, wherein the aggregated frame is associated with a set of cells. The device may obtain an indication of a respective occupancy label for each cell from the set of cells, wherein the respective occupancy label includes a first occupancy label or a second occupancy label, and wherein a subset of cells from the set of cells are associated with the first occupancy label. The device may train, using data associated with the aggregated frame, a machine learning model to generate an occupancy grid, based on a loss function that only calculates a loss for respective cells from the subset of cells. Numerous other aspects are described.

## Claims

### Claim 1

A device, comprising:
one or more memories; and
one or more processors, coupled to the one or more memories, configured to:
receive sensor data associated with a vehicle and a set of frames,
wherein the sensor data indicates one or more sensor detections;
aggregate, using a first pose, the one or more sensor detections into an aggregated frame,
wherein the aggregated frame is associated with a set of cells;
obtain an indication of a respective occupancy label for each cell from the set of cells,
wherein the respective occupancy label includes a first occupancy label indicating a known occupancy status or a second occupancy label indicating an unknown occupancy status, and
wherein a subset of cells from the set of cells are associated with the first occupancy label;
train, using data associated with the aggregated frame, a machine learning model to generate an occupancy grid,
wherein training the machine learning model is associated with a loss function that calculates a loss for respective cells from the subset of cells, and
wherein the machine learning model is trained to predict a probability of an occupancy status for respective cells from the set of cells; and
provide, to another device, the machine learning model.

### Claim 2 (depends on 1)

The device of claim 1, wherein the one or more processors, to train the machine learning model, are configured to:
calculate, using the loss function, the loss for the respective cells from the subset of cells;
refrain from calculating a loss for respective cells associated with the second occupancy label; and
update, based on the loss for the respective cells from the subset of cells, one or more weights associated with the machine learning model.

### Claim 3 (depends on 1)

The device of claim 1, wherein the first occupancy label is associated with indicating the occupancy status of the subset of cells, and
wherein the occupancy status includes a drivable occupancy status or an undrivable occupancy status.

### Claim 4 (depends on 3)

The device of claim 3, wherein the one or more processors, to train the machine learning model, are configured to:
identify, from an output of the machine learning model, one or more cells, from the set of cells, that are associated with an incorrect occupancy status from the drivable occupancy status or the undrivable occupancy status;
apply, for the loss function, a penalty weight for the one or more cells; and
update, based on an output of the loss function, one or more weights associated with the machine learning model.

### Claim 5 (depends on 3)

The device of claim 3, wherein the loss function includes a first loss function associated with the drivable occupancy status and a second loss function associated with the undrivable occupancy status.

### Claim 6 (depends on 1)

The device of claim 1, wherein the one or more processors are further configured to:
convert the aggregated frame from the first pose to a second pose,
wherein the first pose is a common coordinate reference frame for the set of frames, and
wherein the second pose is a coordinate reference frame of a current position of the vehicle.

### Claim 7 (depends on 6)

The device of claim 6, wherein the one or more processors, to train the machine learning model, are configured to:
train the machine learning model using the aggregated frame in the second pose as an input.

### Claim 8 (depends on 1)

The device of claim 1, wherein the data associated with the aggregated frame includes detection data for each cell included in the subset of cells, and wherein the detection data includes at least one of:
one or more radar cross section values,
one or more object velocity values,
a coordinate position of at least one sensor detection,
a quantity of sensor detections, or
an ego velocity value.

### Claim 9

A device, comprising:
one or more memories; and
one or more processors, coupled to the one or more memories, configured to:
receive sensor data associated with a vehicle and a set of frames,
wherein the sensor data indicates one or more sensor detections;
aggregate, using a first pose, the sensor data associated with the set of frames to generate an aggregated frame,
wherein the aggregated frame includes a set of cells, and
wherein the aggregated frame includes data for a subset of cells associated with the one or more sensor detections;
generate, using a machine learning model, an occupancy grid,
wherein the machine learning model is trained using a loss function that calculates a loss using only cells associated with a known occupancy status, and
wherein an output of the machine learning model includes a probability of one or more occupancy statuses for respective cells from the set of cells; and
perform an action based on the occupancy grid.

### Claim 10 (depends on 9)

The device of claim 9, wherein the one or more processors are further configured to:
convert the aggregated frame from the first pose to a second pose,
wherein the first pose is a common coordinate reference frame for the set of frames, and
wherein the second pose is a different coordinate reference frame.

### Claim 11 (depends on 10)

The device of claim 10, wherein the one or more processors, to generate the occupancy grid, are configured to:
generate the occupancy grid using the aggregated frame in the second pose as an input to the machine learning model.

### Claim 12 (depends on 9)

The device of claim 9, wherein the one or more occupancy statuses include a drivable occupancy status or an undrivable occupancy status.

### Claim 13 (depends on 9)

The device of claim 9, wherein the data for the subset of cells includes at least one of:
one or more radar cross section values,
one or more object velocity values,
a coordinate position of at least one sensor detection,
a quantity of sensor detections, or
an ego velocity value.

### Claim 14 (depends on 9)

The device of claim 9, wherein the occupancy grid includes at least one of a static occupancy grid or a dynamic occupancy grid.

### Claim 15 (depends on 9)

The device of claim 9, wherein the sensor data includes at least one of radar data, LIDAR data, or camera data.

### Claim 16

A method, comprising:
receiving, by a device, sensor data associated with a vehicle and a set of frames,
wherein the sensor data indicates one or more sensor detections;
aggregating, using a first pose, the sensor data associated with the set of frames to generate an aggregated frame,
wherein the aggregated frame is associated with a set of cells;
obtaining, by the device, an indication of a respective occupancy label for each cell from the set of cells,
wherein the respective occupancy label includes a first occupancy label indicating a known occupancy status or a second occupancy label indicating an unknown occupancy status, and
wherein a subset of cells from the set of cells are associated with the first occupancy label;
training, using data associated with the aggregated frame, a machine learning model to generate an occupancy grid,
wherein training the machine learning model is associated with a loss function that calculates a loss for respective cells from the subset of cells, and
wherein the machine learning model is trained to predict a probability of an occupancy status for respective cells from the set of cells; and
providing, by the device and to another device, the machine learning model.

### Claim 17 (depends on 16)

The method of claim 16, wherein training the machine learning model comprises:
calculating, using the loss function, the loss for the respective cells from the subset of cells;
refraining from calculating a loss for respective cells associated with the second occupancy label; and
updating, based on the loss for the respective cells from the subset of cells, one or more weights associated with the machine learning model.

### Claim 18 (depends on 16)

The method of claim 16, wherein the first occupancy label is associated with indicating the occupancy status of the subset of cells, and
wherein the occupancy status includes a drivable occupancy status or an undrivable occupancy status.

### Claim 19 (depends on 18)

The method of claim 18, wherein training the machine learning model comprises:
identifying, from an output of the machine learning model, one or more cells, from the set of cells, that are associated with an incorrect occupancy status from the drivable occupancy status or the undrivable occupancy status;
applying, for the loss function, a penalty weight for the one or more cells; and
updating, based on an output of the loss function, one or more weights associated with the machine learning model.

### Claim 20 (depends on 18)

The method of claim 18, wherein the loss function includes a first loss function associated with the drivable occupancy status and a second loss function associated with the undrivable occupancy status.

### Claim 21 (depends on 16)

The method of claim 16, further comprising:
converting the aggregated frame from the first pose to a second pose,
wherein the first pose is a common coordinate reference frame for the set of frames, and
wherein the second pose is a coordinate reference frame of a current position of the vehicle.

### Claim 22 (depends on 16)

The method of claim 16, wherein the occupancy grid includes at least one of a static occupancy grid or a dynamic occupancy grid.

### Claim 23 (depends on 16)

The method of claim 16, wherein the sensor data includes at least one of radar data, LIDAR data, or camera data.

### Claim 24

A method, comprising:
receiving, by a device, sensor data associated with a vehicle and a set of frames,
wherein the sensor data indicates one or more sensor detections;
aggregating, using a first pose, the sensor data associated with the set of frames to generate an aggregated frame,
wherein the aggregated frame includes a set of cells, and
wherein the aggregated frame includes data for a subset of cells associated with the one or more sensor detections;
generating, using a machine learning model, an occupancy grid,
wherein the machine learning model is trained using a loss function that calculates a loss using only cells associated with a known occupancy status, and
wherein an output of the machine learning model includes a probability of one or more occupancy statuses for respective cells from the set of cells; and
performing, by the device, an action based on the occupancy grid.

### Claim 25 (depends on 24)

The method of claim 24, comprising:
obtaining, from another device, the machine learning model.

### Claim 26 (depends on 24)

The method of claim 24, wherein the device is a control unit of the vehicle.

### Claim 27 (depends on 24)

The method of claim 24, further comprising:
converting the aggregated frame from the first pose to a second pose,
wherein the first pose is a common coordinate reference frame for the set of frames, and
wherein the second pose is a different coordinate reference frame.

### Claim 28 (depends on 27)

The method of claim 27, wherein generating the occupancy grid comprises:
generating the occupancy grid using the aggregated frame in the second pose as an input to the machine learning model.

### Claim 29 (depends on 24)

The method of claim 24, wherein the one or more occupancy statuses include a drivable occupancy status or an undrivable occupancy status.

### Claim 30 (depends on 24)

The method of claim 24, wherein the data for the subset of cells includes at least one of:
one or more radar cross section values,
one or more object velocity values,
a coordinate position of at least one sensory detection,
a quantity of sensor detections, or
an ego velocity value.