US 12,576,859 B1Granted

Object classification with out-of-distribution detection

AssigneeZoox, Inc.

InventorsTill Sebastian Hartmann, Daniel Lu Jiang, Nicholas George Dilip Roy

Life of this patent

Estimating the term…

Loading patent…

Abstract

An out-of-distribution (OOD) detector may be provided to efficiently identify OOD inputs to trained machine learning models, to prevent passing misleading predictions from the trained models to downstream systems. During training of a machine learning model including a deep neural network (e.g., an object detection and/or object classifier model based on sensor data), an out-of-distribution detector may be constructed by modeling the density of the intermediate features of the deep neural network (e.g., using a Gaussian mixture model). In some examples, the intermediate features may be within a finetuned model based on a pretrained foundation model, and the machine learning model may be finetuned with regularization toward the original weights of the pretrained model. During inference, the machine learning model may output predictions, such as multilabel object detection and/or classification outputs, and the out-of-distribution detector may be used to efficiently generate confidence values associated with the outputs of the model.

Claims18

Dependent claims

A system comprising:

one or more processors;
and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
receiving sensor data associated with a vehicle in an environment;
providing the sensor data as a first input to a machine learning model comprising a deep neural network configured to determine an object classification based on the first input, wherein the machine learning model is finetuned based on a pretrained model and trained using regularization toward a weight of the pretrained model;
determining, based at least in part on an intermediate layer of the deep neural network, a feature vector associated with the sensor data;
providing the feature vector as a second input to an out-of-distribution detector, wherein the out-of-distribution detector is configured to determine, based at least in part on a Gaussian mixture model (GMM), an indication of whether the first input is an out-of-distribution input for the machine learning model;
and
determining an output of an object classifier associated with the sensor data, based at least in part on a first output of the machine learning model and a second output of the out-of-distribution detector;
and
controlling the vehicle within the environment based at least in part on the output of the object classifier.

8 more elements

A method comprising:

receiving sensor data associated with a vehicle in an environment;
inputting, based at least in part on the sensor data, a first input to a machine learning model, wherein the machine learning model is finetuned based on a pretrained model and trained using regularization toward a weight of the pretrained model;
determining, based at least in part on an intermediate layer of the machine learning model, a feature vector associated with the first input;
determining, based at least in part on a Gaussian mixture model (GMM), a likelihood that the first input is an out-of-distribution input for the machine learning model, wherein the GMM is configured to model density of a vector space associated with the feature vector, and wherein the likelihood comprises a likelihood that the feature vector falls within a distribution of feature vectors associated with a plurality of training data used to train the machine learning model;
and
controlling the vehicle within the environment, based at least in part on the likelihood and on an output of the machine learning model.

3 more elements

One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:

receiving sensor data associated with a vehicle in an environment;
inputting, based at least in part on the sensor data, a first input to a machine learning model;
determining, based at least in part on an intermediate layer of the machine learning model, a feature vector associated with the first input;
determining, based at least in part on the feature vector and a Gaussian mixture model (GMM) configured to model density of a vector space associated with the feature vector, an indication of whether the first input is an out-of-distribution input for the machine learning model;
controlling the vehicle within the environment, based at least in part on the indication and on an output of the machine learning model.

2 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/744,275Filed 14 Jun 2024
Granted: 17 Mar 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification6 codes

B60W 50/0097
Details of control systems for road vehicle drive control not related to the control of a particular sub-unit , e.g. process diagnostic or…
B60W 60/001
Drive control systems specially adapted for autonomous road vehicles: planning or execution of driving tasks
G06F 18/2415
Relating to the classification model, e.g. parametric or non-parametric approaches: based on parametric or probabilistic models, e.g. based on…
B60W 2554/00
Input parameters relating to objects