Temporal-based deformable kernels
AssigneeZoox, Inc.
InventorSarah Tariq
Life of this patent
Estimating the term…
Abstract
Techniques are disclosed for implementing a convolutional neural network that determines an offset field for deforming a kernel to be used in a convolution. The offset field is temporally-based, at least in part, on data generated at an earlier time. Furthermore, techniques are disclosed for using sensor data to train a neural network to learn shapes or configurations of such deformed kernels. The temporal-based deformable convolutions may be used for object identification, object matching, object classification, segmentation, and/or object tracking, in various examples.
Claims21
A system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
- training a machine learned (ML) model to detect a first object in an environment, the training comprising:
- inputting a first image generated at a first time;
- inputting a second image generated at a second time different from the first time;
- determining, based at least in part on the first image and the second image, a tracking velocity associated with a second object, wherein the second object is represented in the first image and the second image, and an optical flow, wherein the tracking velocity and the optical flow are determined on a per-pixel basis between the first image and the second image;
- determining an offset field based at least in part on the tracking velocity and the optical flow;
- applying the offset field to a kernel to produce a modified kernel,
- wherein the kernel is configured to perform two-dimensional convolution and is characterized by a plurality of kernel elements, and applying the offset field comprises displacing an individual kernel element of the plurality of kernel elements in a horizontal, vertical, or diagonal direction;
- further wherein the modified kernel is associated with a pixel of the first image or a pixel of the second image;
- determining, on a per-pixel basis, an auxiliary loss based at least in part on an average offset vector of the offset field and a vector associated with the tracking velocity;
- determining that the auxiliary loss meets or exceeds a threshold;
- updating, based at least in part on the auxiliary loss meeting or exceeding the threshold, the ML model to generate an updated ML model;
- convolving, on a per-pixel basis, the second image with the modified kernel to produce an output;
- and
- transmitting the updated ML model to an autonomous vehicle, wherein detecting an object comprises outputting, by the updated ML model, one or more of a bounding box or an instance segmentation.
A computer-implemented method comprising:
- inputting first data into a machine learning model, the machine learning model trained at least in part by determining an auxiliary loss based at least in part on an average offset vector of an offset field, to one or more of track an object, detect the object, classify the object, or segment data associated with the object in an environment by performing operations comprising:
- inputting second data into the machine learning model;
- generating, for at least a portion of the first data, the offset field, wherein the offset field is based on a difference between the first data and the second data;
- applying the offset field to a kernel;
- generating, based on applying the offset field to the kernel, a modified kernel, and
- convolving, on a per-pixel basis, the portion of the first data with the modified kernel to produce an output feature map.
One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
- inputting first data into a machine learning model trained at least in part by determining an auxiliary loss based at least in part on an average offset vector of an offset field, to one or more of track an object, detect the object, classify the object, or segment data associated with the object in an environment by performing operations comprising:
- generating, for at least a portion of the first data, the offset field, wherein the offset field is based on a difference between the first data and second data;
- applying the offset field to a kernel;
- generating, based on applying the offset field to the kernel, a modified kernel;
- and
- convolving, on a per-pixel basis, the portion of the first data with the modified kernel to produce an output feature map.
Dependencies are inferred from the claim text.