Provisioning real-time three-dimensional maps for autonomous vehicles

A system comprising a processor having access to memory media storing instructions executable by the processor to perform operations comprising:

• receiving, from two or more autonomous vehicles (AVs), respective point cloud data representative of an environment surrounding the AVs;
• receiving a two-dimensional image of the environment;
• at a first processing stage:
• updating a three-dimensional map of the environment based on the received point cloud data;
• extracting three-dimensional proposals of objects of the environment;
• projecting the three-dimensional proposals onto the two-dimensional image of the environment;
• generating, based on the projecting, two-dimensional proposals of the objects of the environment;
• at a second processing stage:
• detecting the objects of the environment from the two-dimensional image of the environment;
• generating, based on the detected objects, two-dimensional bounding boxes of the objects of the environment, each two-dimensional bounding box including semantic information of the object associated with the two-dimensional bounding box;
• comparing the two-dimensional proposals of the objects of the environment with the two-dimensional bounding boxes of the objects of the environment;
• based on the comparing, matching, for one or more of the two-dimensional bounding boxes, the two-dimensional bounding box with a particular two-dimensional proposal of the two-dimensional proposals;
• and
• labeling, for each two-dimensional proposal that is matched to a two-dimensional bounding box, the three-dimensional proposal corresponding to the two-dimensional proposal with the semantic information associated with the two-dimensional bounding box that is matched to the two-dimensional proposal to update the three-dimensional map.

An edge cloud device for provisioning real-time three-dimensional maps, comprising:

• a memory;
• and
• a processor coupled to the memory and configured to include:
• a first processing stage, including:
• a three-dimensional mapping computing module configured to update a three-dimensional map of an environment that includes autonomous vehicles using received point cloud data representative of the environment;
• a connected component computing module configured to extract three-dimensional proposals of objects of the environment;
• and
• a two-dimensional projection computing module configured to i) project the three-dimensional proposals onto a two-dimensional image of the environment, and ii) generate, based on the projecting, two-dimensional proposals of the objects of the environment;
• a second processing stage, including:
• an object detector computing module configured to i) detect the objects of the environment from the two-dimensional image of the environment and ii) generate, based on the detected objects, two-dimensional bounding boxes of the objects of the environment, each two-dimensional bounding box including semantic information of the object associated with the two-dimensional bounding box;
• a multi-sensor fusion (MSF) computing module configured to:
• i) compare the two-dimensional proposals of the objects of the environment with the two-dimensional bounding boxes of the objects of the environment;
• ii) based on the comparing, match, for one or more of the two-dimensional bounding boxes, the two-dimensional bounding box with a particular two-dimensional proposal of the two-dimensional proposals;
• and
• iii) label, for each two-dimensional proposal that is matched to a two-dimensional bounding box, the three-dimensional proposal corresponding to the two-dimensional proposal with the semantic information associated with the two-dimensional bounding box that is matched to the two-dimensional proposal to update the three-dimensional map.