Back to results
US 12,554,266 B1Granted
Sensor anomaly detection using machine-learned attention over spatiotemporally encoded sensor data
AssigneeZoox, Inc.
InventorsMichael Carsten Bosse, Jackson Owen Waschura
Life of this patent
Estimating the term…
Abstract
Detecting a sensor anomaly may include encoding spatiotemporal data associated with one or more sensors.. A first machine-learned model may generate an embedding for a subset of sensor data associated with a ray emanating from a sensor and the embedding may be encoded with spatial characteristics of the sensor that received the subset and a time at which the subset was received. This and other spatiotemporally-encoded embeddings may be used by a second machine-learned model to determine an aggregated embedding that may be used to detect a sensor anomaly.
Claims20
A system comprising:
- one or more processors;
- and
- one or more non-transitory memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
- receiving sensor data from multiple sensors associated with an autonomous vehicle;
- determining, based at least in part on a pose of a first sensor of the multiple sensors, a subset of the sensor data associated with the first sensor and a portion of a field-of-view of the first sensor;
- determining a first feature representation by a first machine-learned model based at least in part on the subset of the sensor data;
- determining, based at least in part on a transformation associated with the pose of the first sensor and a time at which the subset of the sensor data was received, a first transformed feature representation comprising a plurality of transformations of different portions of the first feature representation;
- determining, by a transformer-based machine-learned model based at least in part on cross-attention between the first transformed feature representation and a set of other transformed feature representations, an aggregated feature representation;
- determining, by a second machine-learned model based at least in part on the aggregated feature representation, a sensor anomaly;
- and
- controlling the autonomous vehicle based at least in part on the sensor anomaly.
8 more elements
A method comprising:
- receiving sensor data from multiple sensors associated with an autonomous vehicle;
- determining, based at least in part on a pose of a first sensor of the multiple sensors, a subset of the sensor data associated with the first sensor and a portion of a field-of-view of the first sensor;
- determining a first feature representation by a first machine-learned model based at least in part on the subset of the sensor data;
- determining, based at least in part on a transformation associated with the pose of the first sensor and a time at which the subset of the sensor data was received, a first transformed feature representation comprising a plurality of transformations of different portions of the first feature representation;
- determining, by a transformer-based machine-learned model based at least in part on cross-attention between the first transformed feature representation and a set of other transformed feature representations, an aggregated feature representation;
- determining, by a second machine-learned model based at least in part on the aggregated feature representation, a sensor anomaly;
- and
- controlling the autonomous vehicle based at least in part on the sensor anomaly.
5 more elements
One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, perform operations comprising:
- receiving sensor data from multiple sensors associated with an autonomous vehicle;
- determining, based at least in part on a pose of a first sensor of the multiple sensors, a subset of the sensor data associated with the first sensor and a portion of a field-of-view of the first sensor;
- determining a first feature representation by a first machine-learned model based at least in part on the subset of the sensor data;
- determining, based at least in part on a transformation associated with the pose of the first sensor and a time at which the subset of the sensor data was received, a first transformed feature representation comprising a plurality of transformations of different portions of the first feature representation;
- determining, by a transformer-based machine-learned model based at least in part on cross-attention between the first transformed feature representation and a set of other transformed feature representations, an aggregated feature representation;
- determining, by a second machine-learned model based at least in part on the aggregated feature representation, a sensor anomaly;
- and
- controlling the autonomous vehicle based at least in part on the sensor anomaly.
5 more elements
Dependencies are inferred from the claim text.