Method and apparatus of hierarchical semantic mapping based on deep-learning perception

An apparatus for constructing hierarchical semantic map based on deep-learning based perception, comprising:

• a data acquisition unit, which is configured to obtain 3D point cloud data of a current scene and a scene image of the current scene which are captured by a multi-sensor system, and motion state data of the multi-sensor system during capturing;
• a point cloud projection unit, which is configured to:
• project the 3D point cloud data onto a corresponding image plane to obtain a depth image for the current scene;
• wherein the depth image comprises a plurality of semantic regions, different semantic regions represent different object images, and 3D point cloud data projected in each of the semantic regions is semantically associated with corresponding pixels in the scene image;
• a pose estimation unit, which is configured to perform a point cloud registration process and a pose prediction process on semantically associated 3D point cloud data and the motion state data to obtain an odometer pose related to the multi-sensor system;
• a map construction unit, which is configured to:
• generate a point cloud map for the depth image based on semantically associated 3D point cloud data and the odometer pose of the multi-sensor system, and perform, based on the depth image and the odometer pose of the multi-sensor system, an external ellipse fitting process on each semantic region in the depth image to obtain an object-enveloped map for respective object images;
• and
• a map fusion unit, which is configured to:
• perform a coordinate alignment of the point cloud map and the object-enveloped map according to a configuration of coordinate system extrinsic parameters of the multi-sensor system, and fuse the coordinately aligned point cloud map and the object-enveloped map into a semantic map.