Semantic labeling of point clouds using images

A method comprising:

• obtaining a point cloud, in three spatial dimensions, that is based on lidar data reflecting one or more objects in a space;
• obtaining an image, in two spatial dimensions, that includes a view of at least one of the one or more objects in the space;
• determining a projection of points from the point cloud onto the image;
• generating, using the projection, an augmented image that includes one or more channels of data from the point cloud and one or more channels of data from the image;
• inputting the augmented image to a two dimensional convolutional neural network to obtain a semantic labeled image wherein elements of the semantic labeled image include respective predictions;
• and
• mapping, by reversing the projection, predictions of the semantic labeled image to respective points of the point cloud to obtain a semantic labeled point cloud.

A non-transitory computer-readable storage medium including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations, the operations comprising:

• obtaining a point cloud, in three spatial dimensions, that is based on lidar data reflecting one or more objects in a space;
• obtaining an image, in two spatial dimensions, that includes a view of at least one of the one or more objects in the space;
• determining a projection of points from the point cloud onto the image;
• generating, using the projection, an augmented image that includes one or more channels of data from the point cloud and one or more channels of data from the image;
• inputting the augmented image to a two dimensional convolutional neural network to obtain a semantic labeled image wherein elements of the semantic labeled image include respective predictions;
• and
• mapping, by reversing the projection, predictions of the semantic labeled image to respective points of the point cloud to obtain a semantic labeled point cloud.