Electronic device for generating depth map and operating method thereof

An electronic system comprising:

• a camera configured to obtain a camera image by capturing an ambient environment;
• a Light Detection and Ranging (LIDAR) configured to emit a laser to the ambient environment and to obtain a LIDAR image;
• and
• an electronic device configured to generate a depth map based on the camera image and the LIDAR image,
• wherein the electronic device includes:
• a camera image feature extractor configured to extract a camera image feature based on the camera image;
• a LIDAR image feature extractor configured to extract a LIDAR image feature based on the LIDAR image;
• a sampling unit configured to perform a sampling operation based on the camera image feature and the LIDAR image feature and to generate a sampled LIDAR image feature;
• a fusion unit configured to fuse the camera image feature and the sampled LIDAR image feature and to generate the fusion map;
• and
• a decoding unit configured to decode the fusion map and to generate a depth map,
• wherein the sampling operation includes:
• determining a back-projection ray corresponding to a pixel location of the camera image feature by using intrinsic parameters of the camera;
• generating a plurality of three-dimensional sample points on the back-projection ray in a camera coordinate system;
• transforming the plurality of three-dimensional sample points from the camera coordinate system to a LIDAR coordinate system by using extrinsic parameters between the camera and the LIDAR;
• projecting, using intrinsic parameters of the LIDAR, each of the transformed three-dimensional sample points onto a two-dimensional LIDAR image plane to obtain sampling coordinates;
• and
• sampling the LIDAR image feature at the sampling coordinates to generate the sampled LIDAR image feature.

An operation method of an electronic device which includes a camera image feature extractor, a LIght Detection and Ranging (LIDAR) image feature extractor, a sampling unit, a fusion unit, and a decoding unit, the operation method comprising:

• extracting, at the camera image feature extractor, a camera image feature based on a camera image, wherein the camera image is received from a camera;
• extracting, at the LIDAR image feature extractor, a LIDAR image feature based on a LIDAR image, wherein the LIDAR image is received from a LIDAR;
• performing, at the sampling unit, a sampling operation based on the camera image feature and the LIDAR image feature to generate a sampled LIDAR image feature;
• fusing, at the fusion unit, the camera image feature and the sampled LIDAR image feature to generate a fusion map;
• and
• decoding, at the decoding unit, the fusion map to generate a depth map,
• wherein the sampling operation includes:
• determining a back-projection ray corresponding to a pixel location of the camera image feature by using intrinsic parameters of the camera;
• generating a plurality of three-dimensional sample points on the back-projection ray in a camera coordinate system;
• transforming the plurality of three-dimensional sample points from the camera coordinate system to a LIDAR coordinate system by using extrinsic parameters between the camera and the LIDAR;
• projecting, using intrinsic parameters of the LIDAR, each of the transformed three-dimensional sample points onto a two-dimensional LIDAR image plane to obtain sampling coordinates;
• and
• sampling the LIDAR image feature at the sampling coordinates to generate the sampled LIDAR image feature.