# US 12,008,762 B2 · Systems and methods for generating a road surface semantic segmentation map from a sequence of point clouds

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,762 B2
- Document ID: US12008762B2
- Status: Granted
- Kind: B2
- Application number: 17/676,131
- Filed: 2022-02-19
- Published: 2023-08-24
- Granted: 2024-06-11
- Earliest claimed: 2022-02-19
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Christopher George-R Agia; Ran Cheng; Yuan Ren; Bingbing Liu
- Inventors: Christopher George-R Agia; Ran Cheng; Yuan Ren; Bingbing Liu
- Web page: https://patentgrep.com/patent/US12008762B2

## CPC

- G06T 7/11: Region-based segmentation
- G06V 10/82: using neural networks
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46)
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2431: Multiple classes
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06T 2200/04: involving 3D image data
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06V 2201/12: Acquisition of 3D measurements of objects
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

System and method for semantic segmentation of point clouds. The method may include: generating, via a first neural network, a birds-eye-view (BEV) image of the environment from the aggregated point cloud; generating, via a second neural network, a labelled BEV image from the BEV image, wherein each pixel in the labelled BEV image is associated with a class label from a set of class labels; generating a BEV feature map; and generating, via a third neural network, the road surface segmentation map in the form of a refined labelled BEV image based on the labelled BEV image by smoothing the labelled BEV image using the BEV feature map, wherein each pixel in the refined labelled BEV image is associated with a class label from the set of class labels.

## Claims

### Claim 1

A computer-implemented method of generating a road surface segmentation map, comprising:
receiving a sequence of point clouds, each respective point cloud of the sequence of point clouds representing a three-dimensional (3D) scan of an environment at a different point in time;
generating an aggregated point cloud based on the sequence of point clouds;
generating, via a first neural network, a birds-eye-view (BEV) image of the environment from the aggregated point cloud, wherein the first neural network is a Pillar Feature Net (PFN) neural network;
generating, via a second neural network, a labelled BEV image from the BEV image, wherein each pixel in the labelled BEV image is associated with a class label from a set of class labels, wherein the second neural network is an encoder/decoder (E/D) neural network;
generating a BEV feature map
generating, via a third neural network, the road surface segmentation map in the form of a refined labelled BEV image based on the labelled BEV image by smoothing the labelled BEV image using the BEV feature map, wherein each pixel in the refined labelled BEV image is associated with a class label from the set of class labels, wherein the third neural network is a Convolutional Conditional Random Field (ConvCRF) neural network; and
training the PFN neural network, the E/D neural network and the ConvCRF neural network by, during each training epoch:
generating a plurality of predicted labels by the PFN neural network, the E/D neural network and the ConvCRF neural network;
receiving or retrieving a plurality of ground truth labels;
computing a loss based on the plurality of ground truth labels and the plurality of predicted labels; and
refining one or more weights in the PFN neural network, the E/D neural network and the ConvCRF neural network based on the loss.

### Claim 2 (depends on 1)

The method of claim 1, wherein the BEV feature map is generated based on a set of pillars generated from the aggregated point cloud, wherein each pillar in the set of pillars is a voxel corresponding to a point in the aggregated point cloud with coordinates x, y in the x-y plane and an unlimited spatial extent in the z direction.

### Claim 3 (depends on 2)

The method of claim 2, comprising:
generating the set of pillars generated from the aggregated point cloud.

### Claim 4 (depends on 1)

The method of claim 1, wherein the BEV feature map defines three elements for each pixel in the BEV image, the three elements being each height, intensity and density.

### Claim 5 (depends on 4)

The method of claim 4, wherein the height of a pixel in the BEV feature map represents the difference between points in a pillar having a maximum and a minimum elevation, the intensity of a pixel in the BEV feature map represents a mean of the intensity of the corresponding points in a pillar, and the density of a pixel in the BEV feature map represents the number of points in the respective pillar.

### Claim 6 (depends on 1)

The method of claim 1, wherein the smoothing comprises performing Gaussian kernel smoothing on the labelled BEV image using the BEV feature map, the Gaussian kernel smoothing comprising:
generating Gaussian kernels based on the BEV feature map and the class labels of the labelled BEV image; and
smoothing the labelled BEV image with the Gaussian kernels to generate the refined labelled BEV image.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
generating and storing an elevation value for each pixel in the refined labelled BEV image based on the labelled BEV image.

### Claim 8 (depends on 1)

The method of claim 1, wherein generating the aggregated point cloud is based on an equation:
PC
t
⁢
_
⁢
agg
=
⋂
i
=
1
w
τ
1
,
i
⁢
PC
i
wherein:
PCt_agg represents the aggregated point cloud;
w is a window size parameter;
the point cloud having a most recent timestamp in the sequence of points clouds is a target point cloud;
each point cloud having a timestamp earlier than the most recent timestamp in the sequence of points clouds is a source point cloud;
τ1,i represents a homogenous transformation between the target point cloud and each respective source point cloud i; and
PCi is the respective source point cloud i.

### Claim 9 (depends on 8)

The method of claim 8, wherein the sequence of point clouds is generated by a LIDAR sensor on a vehicle, and the homogenous transformation is computed based on odometry data from the vehicle captured between the target point cloud and each respective source point cloud i.

### Claim 10 (depends on 9)

The method of claim 9, wherein the odometry data comprises data representing a rotation speed of one or more wheels of the vehicle.

### Claim 11 (depends on 8)

The method of claim 8, wherein the value of w is an integer value between 2 to 5.

### Claim 12 (depends on 1)

The method of claim 1, wherein the loss Lsurface is computed based on a local loss term Lfocal and a dice coefficient loss term Ldice based on an equation:
Lsurface=α*Lfocal+(1−α)*Ldice.

### Claim 13 (depends on 12)

The method of claim 12, wherein a has a value between 0.6 and 0.8.

### Claim 14 (depends on 12)

The method of claim 12, wherein the local loss term Lfocal is computed based on an equation:
Lfocal=−μβ(1−{circumflex over (p)}tβ)γlog({circumflex over (p)}tβ),
wherein β the ground truth label for a given pixel, μβ is a class-specific weight, and (1−ptβ)γ is a modulating term.

### Claim 15 (depends on 1)

The method of claim 1, further comprising:
displaying the refined labelled BEV image on a display of a computing system.

### Claim 16

A computing system for generating a road surface segmentation map, the computing system comprising:
a processor configured to:
receive a sequence of point clouds, each respective point cloud of the sequence of point clouds representing a three-dimensional (3D) scan of an environment at a different point in time;
generate an aggregated point cloud based on the sequence of point clouds;
generate, via a first neural network, a birds-eye-view (BEV) image of the environment from the aggregated point cloud, wherein the first neural network is a Pillar Feature Net (PFN) neural network;
generate, via a second neural network, a labelled BEV image from the BEV image, wherein each pixel in the labelled BEV image is associated with a class label from a set of class labels, wherein the second neural network is an encoder/decoder (E/D) neural network;
generate a BEV feature map;
generate, via a third neural network, the road surface segmentation map in the form of a refined labelled BEV image based on the labelled BEV image by smoothing the labelled BEV image using the BEV feature map, wherein each pixel in the refined labelled BEV image is associated with a class label from the set of class labels, wherein the third neural network is a Convolutional Conditional Random Field (ConvCRF) neural network; and
train the PFN neural network, the E/D neural network and the ConvCRF neural network by, during each training epoch:
generating a plurality of predicted labels by the PFN neural network, the E/D neural network and the ConvCRF neural network;
receiving or retrieving a plurality of ground truth labels;
computing a loss based on the plurality of ground truth labels and the plurality of predicted labels; and
refining one or more weights in the PFN neural network, the E/D neural network and the ConvCRF neural network based on the loss.

### Claim 17

A non-transitory machine-readable medium having tangibly stored thereon executable instructions for execution by one or more processors, wherein the executable instructions, in response to execution by the one or more processors, cause the one or more processors to:
receive a sequence of point clouds, each respective point cloud of the sequence of point clouds representing a three-dimensional (3D) scan of an environment at a different point in time;
generate an aggregated point cloud based on the sequence of point clouds;
generate, via a first neural network, a birds-eye-view (BEV) image of the environment from the aggregated point cloud, wherein the first neural network is a Pillar Feature Net (PFN) neural network;
generate, via a second neural network, a labelled BEV image from the BEV image, wherein each pixel in the labelled BEV image is associated with a class label from a set of class labels, wherein the second neural network is an encoder/decoder (E/D) neural network;
generate a BEV feature map;
generate, via a third neural network, the road surface segmentation map in the form of a refined labelled BEV image based on the labelled BEV image by smoothing the labelled BEV image using the BEV feature map, wherein each pixel in the refined labelled BEV image is associated with a class label from the set of class labels, wherein the third neural network is a Convolutional Conditional Random Field (ConvCRF) neural network; and
train the PFN neural network, the E/D neural network and the ConvCRF neural network by, during each training epoch:
generating a plurality of predicted labels by the PFN neural network, the E/D neural network and the ConvCRF neural network;
receiving or retrieving a plurality of ground truth labels;
computing a loss based on the plurality of ground truth labels and the plurality of predicted labels; and
refining one or more weights in the PFN neural network, the E/D neural network and the ConvCRF neural network based on the loss.