Object velocity detection from multi-modal sensor data
AssigneeZoox, Inc.
InventorsSabeek Mani Pradhan, Cooper Stokes Sloan
Life of this patent
Estimating the term…
Abstract
Ground truth data may be too sparse to supervise training of a machine-learned (ML) model enough to achieve an ML model with sufficient accuracy/recall. For example, in some cases, ground truth data may only be available for every third, tenth, or hundredth frame of raw data. Training an ML model to detect a velocity of an object when ground truth data for training is sparse may comprise training the ML model to predict a future position of the object based at least in part on image, radar, and/or lidar data (e.g., for which no ground truth may be available). The ML model may be altered based at least in part on a difference between ground truth data associated with a future time and the future position.
Claims20
A method comprising:
- receiving sensor data associated with a first time, the sensor data comprising point cloud data and image data representing a portion of an environment surrounding an autonomous vehicle;
- receiving an object detection associated with the first time, wherein the object detection identifies an object in the image data;
- determining, based at least in part on the object detection, a first subset of the sensor data comprising a portion of the image data and a portion of the point cloud data;
- inputting the first subset of the sensor data into a machine-learned (ML) model;
- receiving, from the ML model, an output;
- determining, based at least in part on the output, a velocity associated with the object;
- determining, based at least in part on at least one of the output or the velocity, a predicted location of the object at a second time after the first time;
- receiving ground truth data indicative of a three-dimensional ROI associated with the object and the second time;
- determining a difference between the predicted location and a center of the three-dimensional ROI;
- altering one or more parameters of the ML model based at least in part on the difference;
- and
- transmitting the ML model to a vehicle to control motion of the vehicle.
A system comprising:
- one or more processors;
- and
- a memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
- receiving sensor data associated with a first time, the sensor data comprising point cloud data and image data representing a portion of an environment surrounding an autonomous vehicle and the sensor data being associated with an object in the environment;
- inputting the sensor data into a machine-learned (ML) model;
- receiving, from the ML model, an output;
- determining, based at least in part on the output, a velocity associated with the object;
- determining, based at least in part on at least one of the output or the velocity, a predicted location of the object at a second time after the first time;
- receiving ground truth data indicative of a three-dimensional ROI associated with the object and the second time;
- determining a difference between the predicted location and a center of the three-dimensional ROI;
- altering one or more parameters of the ML model based at least in part on the difference;
- and
- transmitting the ML model to a vehicle to control motion of the vehicle.
A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- receiving sensor data associated with a first time, the sensor data comprising point cloud data and image data representing a portion of an environment surrounding an autonomous vehicle and the sensor data being associated with an object in the environment;
- inputting the sensor data into a machine-learned (ML) model;
- receiving, from the ML model, a first three-dimensional region of interest (ROI) and a velocity associated with the object;
- determining, based at least in part on the velocity and the first three-dimensional ROI, a predicted location of the object at a second time after the first time;
- receiving ground truth data indicative of a second three-dimensional ROI associated with the object and the second time;
- determining a difference between the predicted location and a center of the second three-dimensional ROI;
- and altering one or more parameters of the ML model based at least in part on the difference.
Dependencies are inferred from the claim text.