US 12,585,272 B1Granted

Error prioritization for improving machine learned models

AssigneeZoox, Inc.

InventorsMinsu Jang, Antonio Prioletti

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques for determining whether a machine-learned model has improved or regressed in response to an update, as well as verifying whether those improvements or regressions impact overall vehicle safety, are described herein. The techniques may include running a simulation in which a simulated vehicle traverses a simulated environment. During the simulation, sensor data associated with the simulated environment may be input to the machine-learned model, which is configured for use in the real vehicle. As such, perception data outputs associated with objects detected in the simulated environment may be received from the machine-learned model during the simulation, and one or more error models may be generated based on the outputs and a ground truth. If the error model indicates that an error meets or exceeds a threshold error, the machine-learned model may be updated to reduce the error below the threshold.

Claims20

Dependent claims

A system comprising:

one or more processors;
and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
inputting, into a first version of a perception system that is configured for use in an autonomous vehicle, sensor data associated with a simulated environment;
receiving, as outputs from the first version of the perception system, perception data associated with objects detected by a simulated vehicle representing the autonomous vehicle traversing the simulated environment;
generating, based at least in part on the perception data, a first error model indicating at least a first error between outputs of the first version of the perception system and a ground truth associated with the sensor data;
receiving a second error model indicating at least a second error between outputs of a second version of the perception system and the ground truth;
determining, based at least in part on a simulation using a planning component of the autonomous vehicle, that the first error and the second error are of an error type that is associated with contributing to an adverse event;
determining that the first version of the perception system is an improvement over the second version of the perception system based at least in part on a difference between the first error and the second error meeting or exceeding a threshold difference;
and
sending the first version of the perception system to the autonomous vehicle based at least in part on the improvement.

8 more elements

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

inputting, into a machine-learned model that is configured for use in a vehicle, sensor data associated with an environment;
receiving, as outputs from the machine-learned model, perception data associated with objects detected in the environment;
determining, based at least in part on the perception data, an error model including an error between an output of the machine-learned model and a ground truth associated with the sensor data;
determining, based at least in part on comparing the error to a threshold error and a simulation using a planning component, that the error contributes to an adverse event;
based at least in part on the error contributing to the adverse event, determining that the machine-learned model is a regression to the vehicle;
updating the machine-learned model based at least in part on the error;
and
sending, to the vehicle, an updated version of the machine-learned model for use by the vehicle to traverse the environment.

5 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/538,836Filed 30 Nov 2021
Granted: 24 Mar 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification10 codes

G05D 1/0088
Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots: characterized by the…
G06F 30/27
Design optimisation, verification or simulation: using machine learning, e.g. artificial intelligence, neural networks, support vector machines…
G06V 10/774
Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or…:…
G06V 10/776
Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or…:…