# US 12,118,797 B2 · Method for determining a semantic segmentation of an environment of a vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,797 B2
- Document ID: US12118797B2
- Status: Granted
- Kind: B2
- Application number: 17/457,339
- Filed: 2021-12-02
- Published: 2022-06-02
- Granted: 2024-10-15
- Earliest claimed: 2020-12-02
- Assignees: Aptiv Technologies AG
- Applicants: Aptiv Technologies AG
- Inventors: Marco Braun; Moritz Luszek; Jan Siegemund
- Web page: https://patentgrep.com/patent/US12118797B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- B60W 40/04: Traffic conditions
- B60W 40/10: related to vehicle motion
- G01S 7/417: {involving the use of neural networks}
- G01S 13/931: of land vehicles
- G06F 18/24155: {Bayesian classification}
- G06N 3/02: Neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06V 10/82: using neural networks
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2556/40: High definition maps
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method is provided for semantic segmentation of an environment of a vehicle. Via a processing device, a grid of cells is defined dividing the environment of the vehicle. A radar point cloud is received from a plurality of radar sensors, and at least one feature of the radar point cloud is assigned to each grid cell. By using a neural network including deterministic weights, high-level features are extracted for each grid cell. Several classes are defined for the grid cells. For layers of a Bayesian neural network, various sets of weights are determined probabilistically. Via the Bayesian neural network, confidence values are determined for each class and for each grid cell based on the high-level features and based on the various sets of weights in order to determine a predicted class and an extent of uncertainty for the predicted class for each grid cell.

## Claims

### Claim 1

A method comprising:
establishing, with a processing device of a vehicle, a deterministic neural network including deterministic weights and a Bayesian neural network including multiple layers; and
determining, with the processing device, a semantic segmentation of an environment of the vehicle by:
i) executing a feature extracting stage including:
defining a grid of cells dividing the environment of the vehicle;
receiving a radar point cloud from a plurality of radar sensors of the vehicle;
assigning at least one feature of the radar point cloud to each grid cell; and
extracting, using the deterministic neural network, high-level features for each grid cell;
ii) executing a classification stage including:
defining several classes for the grid cells;
for the multiple layers of the Bayesian neural network, probabilistically determining various sets of weights; and
estimating, via the Bayesian neural network, confidence values for each class and for each grid cell based on the high-level features and based on the various sets of weights; and
iii) executing a post processing stage including:
determining, for each grid cell, a predicted class and an extent of uncertainty for the predicted class based on the confidence values,
wherein executing the classification stage ii) comprises:
determining various sets of weights in a predefined number of iterations, each of the iterations providing a respective set of weights and respective confidence values for each class and each grid cell,
determining samples from learned probability distributions for each set of weights of the Bayesian neural network, and
convolving, in each iteration, the high-level features extracted in the feature extraction stage i) with the respective set of weights that are determined by the samples from the learned probability distributions.

### Claim 2 (depends on 1)

The method according to claim 1,
wherein the extent of uncertainty includes an epistemic uncertainty and an aleatoric uncertainty.

### Claim 3 (depends on 2)

The method according to claim 2,
wherein the aleatoric uncertainty is estimated based on an expected entropy of samples from the deterministic and Bayesian neural networks.

### Claim 4 (depends on 3)

The method according to claim 3, wherein executing the post processing stage iii) comprises:
decomposing the uncertainty in the post processing stage; and
estimating a predictive uncertainty based on an entropy of an expected output of the Bayesian neural network, the epistemic uncertainty being estimated as a difference of the predictive uncertainty and the aleatoric uncertainty.

### Claim 5 (depends on 1)

The method according to claim 1, wherein executing the post processing stage iii) comprises:
calculating a mean value over the respective confidence values of the predefined number of iterations for each class and for each grid cell, the highest mean value for each grid cell defining the predicted class of the respective cell.

### Claim 6 (depends on 1)

The method according to claim 1,
wherein the neural network includes a dropout layer and various deterministic layers.

### Claim 7 (depends on 1)

The method according to claim 1,
wherein the at least one feature of the radar point cloud comprises at least one of:
an amount of radar detections per cell, an average Doppler value for each cell, an average distance of each cell to the radar sensors, an average azimuth angle of each cell, or an average radar cross section of radar detections within each cell.

### Claim 8

A system comprising a processing device for a vehicle, the processing device configured to:
establish a deterministic neural network including deterministic weights and a Bayesian neural network including multiple layers; and
determine a semantic segmentation of an environment of the vehicle by:
i) executing a feature extracting stage including:
defining a grid of cells dividing the environment of the vehicle;
receiving a radar point cloud from a plurality of radar sensors of the vehicle;
assigning at least one feature of the radar point cloud to each grid cell; and
extracting, using the deterministic neural network, high-level features for each grid cell;
ii) executing a classification stage including:
defining several classes for the grid cells;
for the multiple layers of the Bayesian neural network, probabilistically determining various sets of weights; and
estimating, via the Bayesian neural network, confidence values for each class and for each grid cell based on the high-level features and based on the various sets of weights; and
iii) executing a post processing stage including:
determining, for each grid cell, a predicted class and an extent of uncertainty for the predicted class based on the confidence values,
wherein the processing device is configured to execute the classification stage ii) by:
determining various sets of weights in a predefined number of iterations, each of the iterations providing a respective set of weights and respective confidence values for each class and each grid cell,
determining samples from learned probability distributions for each set of weights of the Bayesian neural network, and
convolving, in each iteration, the high-level features extracted in the feature extraction stage i) with the respective set of weights that are determined by the samples from the learned probability distributions.

### Claim 9 (depends on 8)

The system according to claim 8,
wherein the extent of uncertainty includes an epistemic uncertainty and an aleatoric uncertainty.

### Claim 10 (depends on 9)

The system according to claim 9,
wherein the aleatoric uncertainty is estimated based on an expected entropy of samples from the deterministic and Bayesian neural networks.

### Claim 11 (depends on 10)

The system according to claim 10, wherein the processing device is configured to execute the post processing stage iii) by:
decomposing the uncertainty in the post processing stage; and
estimating a predictive uncertainty based on an entropy of an expected output of the Bayesian neural network, the epistemic uncertainty being estimated as a difference of the predictive uncertainty and the aleatoric uncertainty.

### Claim 12 (depends on 8)

The system according to claim 8, wherein the processing device is configured to execute the post processing stage iii) by:
calculating a mean value over the respective confidence values of the predefined number of iterations for each class and for each grid cell, the highest mean value for each grid cell defining the predicted class of the respective cell.

### Claim 13

A non-transitory computer readable medium comprising instructions that when executed configure a processing device for a vehicle to:
establish a deterministic neural network including deterministic weights and a Bayesian neural network including multiple layers; and
determine a semantic segmentation of an environment of the vehicle by:
i) executing a feature extracting stage including:
defining a grid of cells dividing the environment of the vehicle;
receiving a radar point cloud from a plurality of radar sensors of the vehicle;
assigning at least one feature of the radar point cloud to each grid cell; and
extracting, using the deterministic neural network, high-level features for each grid cell;
ii) executing a classification stage including:
defining several classes for the grid cells;
for the multiple layers of the Bayesian neural network, probabilistically determining various sets of weights; and
estimating, via the Bayesian neural network, confidence values for each class and for each grid cell based on the high-level features and based on the various sets of weights; and
iii) executing a post processing stage including:
determining, for each grid cell, a predicted class and an extent of uncertainty for the predicted class based on the confidence values,
wherein executing the classification stage ii) comprises:
determining various sets of weights in a predefined number of iterations, each of the iterations providing a respective set of weights and respective confidence values for each class and each grid cell,
determining samples from learned probability distributions for each set of weights of the Bayesian neural network, and
convolving, in each iteration, the high-level features extracted in the feature extraction stage i) with the respective set of weights that are determined by the samples from the learned probability distributions.