Methods and systems for removing objects from view using machine learning

A non-transitory computer readable medium storing instructions executable by a computer processor, the instructions comprising functionality for:

• obtaining a plurality of images from a plurality of cameras, wherein each of the plurality of cameras operates with a fisheye lens and is disposed on a vehicle;
• determining a plurality of depth fields, one for each image in the plurality of images;
• detecting a first distorted object in the plurality of images with a first machine-learned model,
• determining, with the first machine-learned model, a class distribution for the detected first distorted object;
• estimating a distance of the first distorted object from its associated camera using the plurality of depth fields;
• assigning an object weight to the first distorted object based on, at least, the class distribution and the estimated distance;
• removing a first portion from the plurality of images corresponding to the first distorted object in response to the object weight satisfying a criterion;
• generating, with a second machine-learned model, a representative background for the removed first portion based on the plurality of images;
• replacing the first portion in the plurality of images with the representative background;
• stitching and projecting the plurality of images to form an unobstructed bowl view;
• and
• providing the unobstructed bowl view to one or more users of the vehicle, wherein the one or more users are local to the vehicle or remote to the vehicle.

A system, comprising:

• a vehicle;
• a plurality of cameras, wherein each camera in the plurality of cameras is disposed on the vehicle, and wherein each camera in the plurality of cameras is outfitted with a fisheye lens;
• a depth estimator;
• a first machine-learned model;
• a second machine-learned model;
• an image masker;
• and
• a computer, comprising:
• one or more processors, and
• a non-transitory computer readable medium storing instructions executable by the one or more computer processors, the instructions comprising functionality for:
• obtaining a plurality of images from the plurality of cameras;
• determining, with the depth estimator, a plurality of depth fields, one for each image in the plurality of images;
• detecting a first distorted object in the plurality of images with the first machine-learned model;
• determining, with the first machine-learned model a class distribution for the detected first distorted object;
• estimating a distance of the first distorted object from its associated camera using the plurality of depth fields;
• assigning an object weight to the first distorted object based on, at least, the class distribution and estimated distance;
• removing, with the image masker, a first portion from the plurality of images corresponding to the first distorted object in response to the object weight satisfying a criterion;
• generating, with a second machine-learned model, a representative background for the removed first portion based on the plurality of images;
• replacing the first portion in the plurality of images with the representative background;
• stitching and projecting the plurality of images to form an unobstructed bowl view;
• and
• providing the unobstructed bowl view to one or more users of the vehicle, wherein the one or more users are local to the vehicle or remote to the vehicle.