US 2022/0027675 A1 · INSTANCE SEGMENTATION USING SENSOR DATA HAVING DIFFERENT DIMENSIONALITIES

Abstract

Described herein are systems, methods, and non-transitory computer readable media for using 3D point cloud data such as that captured by a LiDAR as ground truth data for training an instance segmentation deep learning model. 3D point cloud data captured by a LiDAR can be projected on a 2D image captured by a camera and provided as input to a 2D instance segmentation model. 2D sparse instance segmentation masks may be generated from the 2D image with the projected 3D data points. These 2D sparse masks can be used to propagate loss during training of the model. Generation and use of the 2D image data with the projected 3D data points as well as the 2D sparse instance segmentation masks for training the instance segmentation model obviates the need to generate and use actual instance segmentation data for training, thereby providing an improved technique for training an instance segmentation model.

Claims20

Dependent claims

A computer-implemented method for training an instance segmentation model using sensor data having different dimensionalities, the method comprising:

capturing first sensor data having a first dimensionality using a first sensor and second sensor data having a second dimensionality using a second sensor;
assigning a first set of labels to the first sensor data to obtain labeled first sensor data and a second set of labels to the second sensor data to obtain labeled second sensor data;
projecting the first sensor data onto the second sensor data to obtain training data;
providing the training data as input to the instance segmentation model;
generating, based at least in part on the training data and the first set of labels, a set of sparse instance segmentation masks having the second dimensionality;
and
utilizing the set of sparse instance segmentation masks to perform loss propagation during training of the instance segmentation model.

4 more elements

A system for training an instance segmentation model using sensor data having different dimensionalities, the system comprising:

at least one processor;
and
at least one memory storing computer-executable instructions, wherein the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:
capture first sensor data having a first dimensionality using a first sensor and second sensor data having a second dimensionality using a second sensor;
assign a first set of labels to the first sensor data to obtain labeled first sensor data and a second set of labels to the second sensor data to obtain labeled second sensor data;
project the first sensor data onto the second sensor data to obtain training data;
provide the training data as input to the instance segmentation model;
generate, based at least in part on the training data and the first set of labels, a set of sparse instance segmentation masks having the second dimensionality;
and
utilize the set of sparse instance segmentation masks to perform loss propagation during training of the instance segmentation model.

7 more elements

Dependencies are inferred from the claim text.

Instance segmentation using sensor data having different dimensionalities

Life of this patent

Instance segmentation using sensor data having different dimensionalities

Life of this patent

Abstract

Claims20