Back to results
US 11,893,750 B2Granted
Multi-task learning for real-time semantic and/or depth aware instance segmentation and/or three-dimensional object bounding
AssigneeZoox, Inc.
InventorsKratarth Goel, James William Vaisey Philbin, Praveen Srinivasan, Sarah Tariq
Life of this patent
Estimating the term…
Abstract
A machine-learning (ML) architecture for determining three or more outputs, such as a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image. The ML architecture may output these outputs at a rate of 30 or more frames per second on consumer grade hardware.
Claims17
A system comprising:
- one or more processors;
- and
- a memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
- receiving an image from an image sensor associated with an autonomous vehicle;
- inputting at least a portion of the image into a machine learned (ML) model;
- determining, by the ML model and based on the image, a set of outputs, the set of outputs comprising:
- a region of interest (ROI) associated with an object that appears in the image;
- a semantic segmentation associated with the object, the semantic segmentation indicative of a classification of the object;
- directional data that indicates a center of the object, wherein a portion of the directional data indicates a direction toward the center of the object from the portion;
- depth data associated with at least the portion of the image, wherein determining the depth data comprises:
- determining, a depth bin from among a set of depth bins, the depth bin associated with a discrete portion of an environment;
- and
- determining a depth residual associated with the depth bin, the depth residual indicating a deviation of a surface associated with the discrete portion from a position associated with the depth bin;
- and
- an instance segmentation associated with the object;
- and
- controlling the autonomous vehicle based at least in part on at least one of the ROI, the semantic segmentation, the instance segmentation, or the depth data.
14 more elements
A method comprising:
- receiving an image from an image sensor associated with an autonomous vehicle;
- inputting at least a portion of the image into a machine learned (ML) model;
- determining, by the ML model and based on the image, a set of outputs, the set of outputs comprising:
- a semantic segmentation associated with an object;
- depth data associated with at least the portion of the image;
- and
- an instance segmentation associated with the object,
- wherein determining the depth data comprises:
- determining, a depth bin from among a set of depth bins, the depth bin associated with a discrete portion of an environment;
- and
- determining a depth residual associated with the depth bin, the depth residual indicating a deviation of a surface associated with the discrete portion from a position associated with the depth bin;
- and
- controlling the autonomous vehicle based at least in part on at least one of the semantic segmentation, the instance segmentation, or the depth data.
10 more elements
A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- receiving an image from an image sensor associated with an autonomous vehicle;
- inputting at least a portion of the image into a machine learned (ML) model;
- determining, by the ML model and based on the image, a set of outputs, the set of outputs comprising:
- a semantic segmentation associated with an object;
- depth data associated with at least the portion of the image, wherein determining the depth data comprises:
- determining, a depth bin from among a set of depth bins, the depth bin associated with a discrete portion of an environment;
- and
- determining a depth residual associated with the depth bin, the depth residual indicating a deviation of a surface associated with the discrete portion from a position associated with the depth bin;
- directional data that indicates a center of the object, wherein a portion of the directional data indicates a direction toward the center of the object from the portion;
- and
- an instance segmentation associated with the object;
- and
- controlling the autonomous vehicle based at least in part on at least one of the semantic segmentation, the instance segmentation, or the depth data.
10 more elements
Dependencies are inferred from the claim text.