Generating occlusion-aware bird eye view representations of complex road scenes

An autonomous vehicle for autonomous navigation with an occlusion-aware bird's eye view map of a road scene, the vehicle comprising:

• an image capture device that captures an image of a road scene with background objects and foreground objects;
• a semantic segmentation network that identifies foreground objects and background objects in the image by extracting foreground features corresponding to the foreground objects and background features corresponding to the background objects, the foreground object occluding the background objects from a view of a camera that captured the image;
• a masking network that masks the foreground objects from the image with a mask to generate a masked image;
• a semantic in-painting network that infers occluded objects by predicting semantic features in masked areas of the masked image according to contextual information related to the identified background features visible in the masked image;
• a depth in-painting network that infers depths of the occluded objects by predicting depths in masked areas of the masked image according to the contextual information;
• a background mapping system that maps the foreground objects and the background objects to a three-dimensional space according to locations of each of the foreground objects, the background objects and occluded object using the inferred depths;
• a view converter that generates a bird's eye view from the three-dimensional space;
• and
• a control unit that adjusts a steering, a throttle and one or more brakes of the vehicle to navigate roads of the road scene while avoiding collisions.