Back to results
US 11,958,501 B1Granted
Performance-based metrics for evaluating system quality
AssigneeZoox, Inc.
InventorPeter Scott Schleede
Life of this patent
Estimating the term…
Abstract
Techniques relating to performance-based metrics for evaluating system quality are described. In an example, sensor data associated with an environment within which a vehicle is positioned is received. A performance metric associated with a trajectory, indicative of a performed behavior of the vehicle, can be determined and a correctness metric can be determined based at least in part on the performance metric. The correctness metric can represent a correctness of the performed behavior. Modification(s) to component(s) and/or system(s), or portions thereof, of the vehicle can be affected based at least in part on the correctness metric.
Claims20
A method comprising:
- receiving sensor data associated with an environment within which a vehicle is positioned;
- detecting, based at least in part on the sensor data, an object proximate the vehicle in the environment;
- generating, in response to detecting the object, a first trajectory and a second trajectory for controlling the vehicle relative to the object;
- determining, based at least in part on the first trajectory, a first performance metric determined as a combination of a first metric associated with an amount of time for the vehicle to progress to a desired state after executing the first trajectory and a second metric associated with a proximity of the vehicle to the object along the first trajectory;
- determining, based at least in part on the second trajectory, a second performance metric;
- determining, based at least in part on the first performance metric and the second performance metric, a correctness metric that represents a correctness of a performed behavior of the vehicle;
- determining a plurality of combined metrics, wherein individual combined metrics of the plurality of combined metrics for the first trajectory and the second trajectory are determined based on the first performance metric, the second performance metric, and the correctness metric;
- determining a period of time in which one or more of the plurality of combined metrics are below a threshold;
- and
- updating a component of the vehicle based at least in part on accessing data associated with the period of time,
- wherein the correctness metric is associated with a difference between the first trajectory and the second trajectory.
8 more elements
A system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
- receiving sensor data associated with an environment within which a vehicle is positioned;
- detecting, based at least in part on the sensor data, an object proximate the vehicle;
- determining, for individual instances of time and based at least in part on a velocity of the vehicle and a proximity of the vehicle relative to the object, a plurality of performance metrics, individual performance metrics of the plurality of performance metrics being associated with a trajectory and indicative of a performed behavior of the vehicle, wherein the individual performance metrics are based at least in part on a first metric associated with an amount of time for the vehicle to progress to a desired state after executing the trajectory and a second metric associated with a proximity of the vehicle to the object along the trajectory;
- determining, for the individual instances of time, a plurality of trajectories for controlling the vehicle proximate the object, wherein the trajectory comprises one of the plurality of trajectories;
- determining a correctness metric based at least in part on the plurality performance metrics, wherein the correctness metric represents a correctness of the performed behavior, and the correctness metric is associated with a difference between the plurality of trajectories;
- determining a plurality of combined metrics, wherein individual combined metrics of the plurality of combined metrics for the trajectory are determined based on the individual performance metrics of the plurality of performance metrics and the correctness metric;
- determining a period of time in which one or more of the plurality of combined metrics is below a threshold value;
- updating a component of the vehicle based at least in part on accessing data associated with the period of time;
- and
- controlling the vehicle based at least in part on the correctness metric.
10 more elements
One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- receiving sensor data associated with an environment within which a vehicle is positioned;
- determining, at individual instances of time, a plurality of performance metrics associated with a plurality of trajectories, individual performance metrics of the plurality of performance metrics being indicative of a performed behavior of the vehicle and based at least in part on one or more of a speed of the vehicle or a proximity of the vehicle to an object proximate the vehicle, wherein the individual performance metrics are determined based at least in part on a first metric proportional to an amount of time for the vehicle to progress to a desired state after executing a trajectory of the plurality of trajectories and a second metric proportional to the proximity of the vehicle to the object along the trajectory;
- determining a function based at least in part on the plurality of performance metrics;
- determining a correctness metric based at least in part on the function, wherein the correctness metric represents a correctness of the performed behavior, and the correctness metric is associated with a difference between the plurality of trajectories;
- determining a plurality of combined metrics, wherein individual combined metrics of the plurality of combined metrics for the trajectory are determined based on a performance metric of the plurality of performance metrics and the correctness metric;
- determining a period of time in which one or more of the plurality of combined metrics is below a threshold;
- updating a component of the vehicle based at least in part on accessing data associated with the period of time;
- and
- outputting a signal based at least in part on the correctness metric.
6 more elements
Dependencies are inferred from the claim text.