Back to results
US 10,832,062 B1Granted
Image embedding for object tracking
AssigneeZoox, Inc.
InventorsBryce A. Evans, James William Vaisey Philbin, Sarah Tariq
Life of this patent
Estimating the term…
Abstract
Techniques are disclosed for implementing a neural network that outputs embeddings. Furthermore, techniques are disclosed for using sensor data to train a neural network to learn such embeddings. In some examples, the neural network may be trained to learn embeddings. The embeddings may be used for object identification, object matching, object classification, and/or object tracking in various examples.
Claims20
A system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
- tracking, at least partly via one or more cameras of an autonomous vehicle, one or more objects associated with an environment of the autonomous vehicle, wherein the tracking comprises:
- receiving a first image comprising a representation of a first object in the environment;
- determining a first image patch of the first image, the first image patch comprising at least a portion of the first image associated with the first object;
- inputting the first image patch into a neural network trained to determine an embedding;
- receiving, from the neural network and based at least in part on the first image patch, a first output associated with a first embedding, the first embedding comprising a first vector descriptive of the first object;
- receiving a second image comprising a representation of a second object in the environment;
- determining a second image patch of the second image, the second image patch comprising at least a portion of the second image associated with the second object;
- inputting the second image patch into the neural network;
- receiving, from the neural network, a second output associated with a second embedding, the second embedding comprising a second vector descriptive of the second object;
- determining a distance between the first embedding and the second embedding;
- and
- determining, based at least in part on the distance, whether the first object associated with the first embedding and represented in the first image corresponds to the second object associated with the second embedding and represented in the second image;
- and
- controlling, based at least in part on the tracking, movement of the autonomous vehicle.
14 more elements
A computer-implemented method comprising:
- tracking one or more objects, wherein the tracking comprises:
- receiving, from a first sensor of a vehicle, first data;
- determining, based at least in part on the first data, a first subset of data;
- inputting the first subset of data into a neural network trained to determine an n-dimensional representation of the first subset of data, wherein the first subset of data is associated with a first object;
- receiving, from the neural network and based at least in part on the first subset of data, a first n-dimensional representation;
- receiving, from a second sensor of the vehicle, second data;
- determining, based at least in part on the second data, a second subset of data;
- inputting the second subset of data into the neural network, wherein the second subset of data is associated with a second object;
- receiving, from the neural network and based at least in part on the second subset of data, a second n-dimensional representation;
- determining, based at least in part on a distance between the first n-dimensional representation and the second n-dimensional representation, that the second object corresponds to the first object;
- and
- determining, based at least in part on the second object represented in the second data corresponding to the first object represented in the first data, a velocity and an orientation of the first object;
- and
- controlling movement of the vehicle based at least in part on one or more of the velocity or the orientation of the first object.
11 more elements
One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
- tracking one or more objects, wherein the tracking comprises:
- receiving, from a sensor of a vehicle, first data;
- determining, based at least in part on the first data, a first subset of data;
- inputting the first subset of data into a neural network that determines n-dimensional representations of input data, wherein the first subset of data is associated with a first object;
- receiving, from the neural network and based at least in part on the first subset of data, an output associated with a first n-dimensional representation;
- receiving, from the sensor of the vehicle, second data;
- determining, based at least in part on the second data, a second subset of data;
- inputting the second subset of data into the neural network, wherein the second subset of data is associated with a second object;
- receiving, from the neural network and based at least in part on the second subset of data, a second n-dimensional representation;
- determining, based at least in part on a distance between the first n-dimensional representation and the second n-dimensional representation, that the second object represented in the second data corresponds to the first object represented in the first data;
- and
- determining, based at least in part on the second object corresponding to the first object, one or more of a velocity or an orientation of the first object;
- and
- controlling movement of the vehicle based at least in part on one or more of the velocity of the first object or the orientation of the first object.
11 more elements
Dependencies are inferred from the claim text.