Refining depth from an image
AssigneeZoox, Inc.
InventorPraveen Srinivasan
Life of this patent
Estimating the term…
Abstract
Depth estimates for an object made by one or more sensors of a vehicle may be refined using locations of environmental attributes that are proximate the object. An image captured of the object proximate an environmental attribute may be analyzed to determine where the object is positioned relative to the environmental attribute. A machine-learned model may be used to detect the environmental attribute, and a location of the environmental attribute may be determined from map data. A probability of a location of the object may be determined based on the known location of the environmental attribute. The location probability of the object may be used to refine depth estimates generated by other means, such as a monocular depth estimation from an image using computer vision.
Claims20
A system comprising:
- one or more communication interfaces;
- one or more processors;
- and
- one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
- receiving, from a sensor of an autonomous vehicle, an image depicting an object in an environment;
- determining, based at least in part on the image, a first depth of the object;
- inputting at least a portion of the image associated with the object into a machine-learned model trained to output an association of one or more environmental attributes with objects represented in images input into the machine-learned model;
- receiving, from the machine-learned model, an output indicative of an environmental attribute associated with the object;
- receiving, at least in part via the one or more communication interfaces, map data associated with the environmental attribute, the map data being stored in memory and including a previously known location of the environmental attribute;
- determining, based at least in part on the map data, a first location of the environmental attribute in the environment;
- determining, based at least in part on the output and the first location of the environmental attribute in the environment, a second depth associated with a second location of the object in the environment;
- and
- determining, based at least in part on the first depth and the second depth associated with the object in the environment, an estimated depth of the object.
A method comprising:
- receiving image data depicting an object in an environment;
- determining, based at least in part on the image data, a first distance to the object;
- inputting at least a portion of the image data associated with the object into a machine-learned model;
- receiving, from the machine-learned model, an output indicative of an environmental attribute associated with the object;
- receiving, at least partially via a communication interface, stored map data associated with the environmental attribute, the stored map data being stored in memory and including a previously known location of the environmental attribute;
- determining, based at least in part on the stored map data, a first location of the environmental attribute in the environment;
- determining, based at least in part on the output and the first location of the environmental attribute in the environment, a second distance to the object;
- and
- determining, based at least in part on the first distance and the second distance, a refined distance of the object.
One or more non-transitory computer-readable media storing instructions that when executed by one or more processors perform operations comprising:
- determining a first depth of an object represented in image data captured by a sensor of a vehicle;
- inputting at least a portion of the image data associated with the object into a machine-learned model;
- receiving, from the machine-learned model, an output indicative of an environmental attribute associated with the object;
- receiving map data associated with the environmental attribute, the map data being stored in memory and including a previously known location of the environmental attribute;
- determining, based at least in part on the map data, a first location of the environmental attribute in an environment;
- determining, based at least in part on the output indicative of the environmental attribute received from the machine-learned model and the location of the environmental attribute in the environment, a second depth of the object;
- and
- determining, based at least in part on the first depth and the second depth, a refined depth of the object.
Dependencies are inferred from the claim text.