Systems and methods for generating a road surface semantic segmentation map from a sequence of point clouds

A computing system for generating a road surface segmentation map, the computing system comprising:

• a processor configured to:
• receive a sequence of point clouds, each respective point cloud of the sequence of point clouds representing a three-dimensional (3D) scan of an environment at a different point in time;
• generate an aggregated point cloud based on the sequence of point clouds;
• generate, via a first neural network, a birds-eye-view (BEV) image of the environment from the aggregated point cloud, wherein the first neural network is a Pillar Feature Net (PFN) neural network;
• generate, via a second neural network, a labelled BEV image from the BEV image, wherein each pixel in the labelled BEV image is associated with a class label from a set of class labels, wherein the second neural network is an encoder/decoder (E/D) neural network;
• generate a BEV feature map;
• generate, via a third neural network, the road surface segmentation map in the form of a refined labelled BEV image based on the labelled BEV image by smoothing the labelled BEV image using the BEV feature map, wherein each pixel in the refined labelled BEV image is associated with a class label from the set of class labels, wherein the third neural network is a Convolutional Conditional Random Field (ConvCRF) neural network;
• and
• train the PFN neural network, the E/D neural network and the ConvCRF neural network by, during each training epoch:
• generating a plurality of predicted labels by the PFN neural network, the E/D neural network and the ConvCRF neural network;
• receiving or retrieving a plurality of ground truth labels;
• computing a loss based on the plurality of ground truth labels and the plurality of predicted labels;
• and
• refining one or more weights in the PFN neural network, the E/D neural network and the ConvCRF neural network based on the loss.

A non-transitory machine-readable medium having tangibly stored thereon executable instructions for execution by one or more processors, wherein the executable instructions, in response to execution by the one or more processors, cause the one or more processors to:

• receive a sequence of point clouds, each respective point cloud of the sequence of point clouds representing a three-dimensional (3D) scan of an environment at a different point in time;
• generate an aggregated point cloud based on the sequence of point clouds;
• generate, via a first neural network, a birds-eye-view (BEV) image of the environment from the aggregated point cloud, wherein the first neural network is a Pillar Feature Net (PFN) neural network;
• generate, via a second neural network, a labelled BEV image from the BEV image, wherein each pixel in the labelled BEV image is associated with a class label from a set of class labels, wherein the second neural network is an encoder/decoder (E/D) neural network;
• generate a BEV feature map;
• generate, via a third neural network, the road surface segmentation map in the form of a refined labelled BEV image based on the labelled BEV image by smoothing the labelled BEV image using the BEV feature map, wherein each pixel in the refined labelled BEV image is associated with a class label from the set of class labels, wherein the third neural network is a Convolutional Conditional Random Field (ConvCRF) neural network;
• and
• train the PFN neural network, the E/D neural network and the ConvCRF neural network by, during each training epoch:
• generating a plurality of predicted labels by the PFN neural network, the E/D neural network and the ConvCRF neural network;
• receiving or retrieving a plurality of ground truth labels;
• computing a loss based on the plurality of ground truth labels and the plurality of predicted labels;
• and
• refining one or more weights in the PFN neural network, the E/D neural network and the ConvCRF neural network based on the loss.