High resolution 3D point clouds generation from downsampled low resolution LIDAR 3D point clouds and camera images

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:

• receiving a first image captured by a first camera, the first image capturing a portion of a driving environment of an autonomous driving vehicle (ADV);
• receiving a second image representing a first depth map of a first point cloud corresponding to the portion of the driving environment produced by a light detection and ranging (LiDAR) device;
• downsampling the second image by a predetermined scale factor until a resolution of the second image reaches a predetermined threshold;
• and
• generating a second depth map by applying a convolutional neural network (CNN) model to the first image and the downsampled second image, the second depth map having a higher resolution than the first depth map, wherein the second depth map represents a second point cloud perceiving the driving environment surrounding the ADV.

A data processing system, comprising:

• a processor;
• and
• a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
• receiving a first image captured by a first camera, the first image capturing a portion of a driving environment of an autonomous driving vehicle (ADV);
• receiving a second image representing a first depth map of a first point cloud corresponding to the portion of the driving environment produced by a light detection and ranging (LiDAR) device;
• downsampling the second image by a predetermined scale factor until a resolution of the second image reaches a predetermined threshold;
• and
• generating a second depth map by applying a convolutional neural network (CNN) model to the first image and the downsampled second image, the second depth map having a higher resolution than the first depth map, wherein the second depth map represents a second point cloud perceiving the driving environment surrounding the ADV.