# US 12,183,047 B2 · Methods and systems for removing objects from view using machine learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,047 B2 - Document ID: US12183047B2 - Status: Granted - Kind: B2 - Application number: 18/048,649 - Filed: 2022-10-21 - Published: 2024-04-25 - Granted: 2024-12-31 - Earliest claimed: 2022-10-21 - Assignees: Valeo Schalter und Sensoren GmbH - Applicants: Valeo Schalter und Sensoren GmbH - Inventors: Jagdish Bhanushali; Peter Groth - Web page: https://patentgrep.com/patent/US12183047B2 ## CPC - G06V 10/16: {using multiple overlapping images; Image stitching} - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 7/55: from multiple images - G06V 10/273: {removing elements interfering with the pattern to be recognised} - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/778: Active pattern-learning, e.g. online learning of image or video features - G06V 10/82: using neural networks - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60R 1/22: for viewing an area outside the vehicle, e.g. the exterior of the vehicle - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06V 2201/08: Detecting or categorising vehicles - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR ## Abstract A method for generating an unobstructed bowl view of a vehicle that includes obtaining a plurality of images from a plurality of cameras disposed on the vehicle and determining a plurality of depth fields. The method further includes detecting a plurality of distorted objects in the plurality of images with a first machine-learned model that assigns a class distribution to the detected distorted object and estimating a distance of each distorted object from its associated camera using the plurality of depth fields. The method further includes assigning an object weight to each distorted object in the plurality of distorted objects and removing at least one distorted objects from the plurality of images. The method further includes replacing each of the at least one removed distorted objects with a representative background generated by a second machine-learned model and stitching and projecting the plurality of images to form the unobstructed bowl view. ## Claims ### Claim 1 A method for generating an unobstructed bowl view of a vehicle, comprising: obtaining a plurality of images from a plurality of cameras, wherein each of the plurality of cameras operates with a fisheye lens and is disposed on the vehicle; determining a plurality of depth fields, one for each image in the plurality of images; detecting a first distorted object in the plurality of images with a first machine-learned model, determining, with the first machine-learned model, a class distribution for the detected first distorted object; estimating a distance of the first distorted object from its associated camera using the plurality of depth fields; assigning an object weight to the first distorted object based on, at least, the class distribution and the estimated distance; removing a first portion from the plurality of images corresponding to the first distorted object in response to the object weight satisfying a criterion; generating, with a second machine-learned model, a representative background for the removed first portion based on the plurality of images; replacing the first portion in the plurality of images with the representative background; stitching and projecting the plurality of images to form the unobstructed bowl view; and providing the unobstructed bowl view to one or more users of the vehicle, wherein the one or more users are local to the vehicle or remote to the vehicle. ### Claim 2 (depends on 1) The method of claim 1, further comprising: pre-processing the plurality of images. ### Claim 3 (depends on 1) The method of claim 1, further comprising: blending the representative background and a surrounding portion in the plurality of images, wherein the surrounding portion excludes the first portion. ### Claim 4 (depends on 1) The method of claim 1, further comprising: training the first machine-learned model with a first training dataset, wherein the first training dataset comprises training images with annotated distorted objects obtained with one or more cameras outfitted with a fisheye lens. ### Claim 5 (depends on 1) The method of claim 1, further comprising: training a generative adversarial network comprising a generator and a discriminator, wherein the second machine-learned model is the generator, wherein the generator is trained with a second training dataset comprising masked images wherein, for each masked image, a masked portion is retained as a correct background, wherein the discriminator is trained with a third training dataset comprising generated images, generated by the generator, and original images. ### Claim 6 (depends on 1) The method of claim 1, wherein the first machine-learned model is a convolutional neural network. ### Claim 7 (depends on 1) The method of claim 1, wherein the second machine-learned model is trained by optimizing a loss function, wherein the loss function uses, in part, the estimated depth and class distribution of an annotated distorted object. ### Claim 8 A non-transitory computer readable medium storing instructions executable by a computer processor, the instructions comprising functionality for: obtaining a plurality of images from a plurality of cameras, wherein each of the plurality of cameras operates with a fisheye lens and is disposed on a vehicle; determining a plurality of depth fields, one for each image in the plurality of images; detecting a first distorted object in the plurality of images with a first machine-learned model, determining, with the first machine-learned model, a class distribution for the detected first distorted object; estimating a distance of the first distorted object from its associated camera using the plurality of depth fields; assigning an object weight to the first distorted object based on, at least, the class distribution and the estimated distance; removing a first portion from the plurality of images corresponding to the first distorted object in response to the object weight satisfying a criterion; generating, with a second machine-learned model, a representative background for the removed first portion based on the plurality of images; replacing the first portion in the plurality of images with the representative background; stitching and projecting the plurality of images to form an unobstructed bowl view; and providing the unobstructed bowl view to one or more users of the vehicle, wherein the one or more users are local to the vehicle or remote to the vehicle. ### Claim 9 (depends on 8) The non-transitory computer readable medium of claim 8, further comprising instructions for: pre-processing the plurality of images. ### Claim 10 (depends on 8) The non-transitory computer readable medium of claim 8, further comprising instructions for: blending the representative background and a surrounding portion in the plurality of images, wherein the surrounding portion excludes the first portion. ### Claim 11 (depends on 8) The non-transitory computer readable medium of claim 8, further comprising instructions for: training the first machine-learned model with a first training dataset, wherein the first training dataset comprises training images with annotated distorted objects obtained with one or more cameras outfitted with a fisheye lens. ### Claim 12 (depends on 8) The non-transitory computer readable medium of claim 8, further comprising instructions for: training a generative adversarial network comprising a generator and a discriminator, wherein the second machine-learned model is the generator, wherein the generator is trained with a second training dataset comprising masked images wherein, for each masked image, a masked portion is retained as a correct background, wherein the discriminator is trained with a third training dataset comprising generated images, generated by the generator, and original images. ### Claim 13 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the first machine-learned model is a convolutional neural network. ### Claim 14 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the second machine-learned model is trained by optimizing a loss function, wherein the loss function uses, in part, the estimated depth and class distribution of an annotated distorted object. ### Claim 15 A system, comprising: a vehicle; a plurality of cameras, wherein each camera in the plurality of cameras is disposed on the vehicle, and wherein each camera in the plurality of cameras is outfitted with a fisheye lens; a depth estimator; a first machine-learned model; a second machine-learned model; an image masker; and a computer, comprising: one or more processors, and a non-transitory computer readable medium storing instructions executable by the one or more computer processors, the instructions comprising functionality for: obtaining a plurality of images from the plurality of cameras; determining, with the depth estimator, a plurality of depth fields, one for each image in the plurality of images; detecting a first distorted object in the plurality of images with the first machine-learned model; determining, with the first machine-learned model a class distribution for the detected first distorted object; estimating a distance of the first distorted object from its associated camera using the plurality of depth fields; assigning an object weight to the first distorted object based on, at least, the class distribution and estimated distance; removing, with the image masker, a first portion from the plurality of images corresponding to the first distorted object in response to the object weight satisfying a criterion; generating, with a second machine-learned model, a representative background for the removed first portion based on the plurality of images; replacing the first portion in the plurality of images with the representative background; stitching and projecting the plurality of images to form an unobstructed bowl view; and providing the unobstructed bowl view to one or more users of the vehicle, wherein the one or more users are local to the vehicle or remote to the vehicle. ### Claim 16 (depends on 15) The system of claim 15, wherein the computer further comprises instructions for: pre-processing the plurality of images. ### Claim 17 (depends on 15) The system of claim 15, wherein the computer further comprises instructions for: blending the representative background and a surrounding portion in the plurality of images, wherein the surrounding portion excludes the first portion. ### Claim 18 (depends on 15) The system of claim 15, wherein the computer further comprises instructions for: training the first machine-learned model with a first training dataset, wherein the first training dataset comprises training images with annotated distorted objects obtained with one or more cameras outfitted with a fisheye lens. ### Claim 19 (depends on 15) The system of claim 15, wherein the computer further comprises instructions for: training a generative adversarial network comprising a generator and a discriminator, wherein the second machine-learned model is the generator, wherein the generative adversarial network is trained with a second training dataset comprising masked images wherein, for each masked image, a masked portion is retained as a correct background, wherein the discriminator is trained with a third training dataset comprising generated images, generated by the generator, and original images. ### Claim 20 (depends on 15) The system of claim 15, wherein the second machine-learned model is trained by optimizing a loss function, wherein the loss function uses, in part, the estimated depth and class distribution of an annotated distorted object.