Back to results
US 12,691,906 B1Granted
Object detection and tracking, including parallelized lidar semantic segmentation
AssigneeZoox, Inc.
InventorsSubhasis Das, Benjamin Isaac Zwiebel
Life of this patent
Estimating the term…
Abstract
Generating a semantic segmentation of point cloud data may include detecting an object based at least in part one or more images and using the image-based object detection in combination with the point cloud data to determine the semantic segmentation of the point cloud data. The semantic segmentation of point cloud data may be conducted in parallel with executing an object tracking component. Parallelizing these functions reduces the amount of time it takes to use sensors to detect and track objects and the point cloud semantic segmentation may provide further granularity for planning and ML model training.
Claims20
A method comprising:
- receiving sensor data, wherein the sensor data comprises image data and lidar data comprising three-dimensional lidar points;
- determining, based on the image data, an object detection indicating:
- a portion of the image data that is associated with an object in an environment being traversed by a vehicle, and
- a first object classification associated with the portion;
- determining, based on projecting a first lidar point of the three-dimensional lidar points to a two-dimensional space associated with the image data, a projected lidar point in the two-dimensional space;
- determining, based at least in part on whether the projected lidar point falls within the portion of the image data, a probability that the first lidar point is associated with the first object classification;
- determining, based at least in part on the probability meeting or exceeding a probability threshold, that the first lidar point is included in a subset of the three-dimensional lidar points that is associated with the first object classification,
- wherein the subset of the three-dimensional lidar points is determined based on projecting the subset to the two-dimensional space associated with the image data;
- determining a three-dimensional region of interest (ROI) that includes a threshold portion of the subset of the three-dimensional lidar points;
- and
- controlling the vehicle based at least in part on the three-dimensional ROI.
8 more elements
A system comprising:
- one or more processors;
- and
- a memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
- receiving sensor data, wherein the sensor data comprises image data and point cloud data;
- receiving an object detection associated with an object in an environment being traversed by a vehicle, the object detection indicating:
- a portion of the image data associated with the object, and
- a first object classification associated with the object;
- determining, based on projecting a first point of the point cloud data, from a three-dimensional space associated with the point cloud data into a two-dimensional space associated with the image data, a probability that the first point is associated with the first object classification;
- determining, based at least in part on the probability meeting or exceeding a probability threshold, that the first point is included in a subset of points of the point cloud data that is associated with the first object classification,
- wherein the subset of the points is determined based on projecting the subset to the two-dimensional space associated with the image data;
- determining a three-dimensional region of interest (ROI) that includes a threshold portion of the subset of the points;
- and
- controlling the vehicle based at least in part on the three-dimensional ROI.
10 more elements
A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- receiving sensor data, wherein the sensor data comprises image data and point cloud data representing an environment associated with a vehicle;
- determining, an object detection indicating a classification of an object in the environment and a portion of the image data associated with the object, wherein the classification includes an identification of an object type;
- determining, based on projecting a first point of the point cloud data into a two-dimensional space associated with the image data, a probability that the first point is associated with the classification;
- determining, based at least in part on the probability meeting or exceeding a probability threshold, that the first point is included in a subset of points of the point cloud data that is associated with the classification,
- wherein the subset of the points is determined based on projecting the subset to the two-dimensional space associated with the image data;
- determining a three-dimensional region of interest (ROI) that includes a threshold portion of the subset of the points;
- and
- controlling the vehicle based at least in part on the three-dimensional ROI.
5 more elements
Dependencies are inferred from the claim text.