Back to results
US 11,887,363 B2Granted
Training a deep neural network model to generate rich object-centric embeddings of robotic vision data
AssigneeGoogle Inc.
InventorsSoeren Pirk, Yunfei Bai, Pierre Sermanet, Seyed Mohammad Khansari Zadeh, Harrison Lynch
Life of this patent
Estimating the term…
Abstract
Training a machine learning model (e.g., a neural network model such as a convolutional neural network (CNN) model) so that, when trained, the model can be utilized in processing vision data (e.g., from a vision component of a robot), that captures an object, to generate a rich object-centric embedding for the vision data. The generated embedding can enable differentiation of even subtle variations of attributes of the object captured by the vision data.
Claims19
A method implemented by one or more processors, the method comprising:
- identifying a first image of an environment, the first image capturing objects in the environment and being captured from a first point of view;
- identifying a second image of the environment, the second image capturing at least a subset of the objects in the environment and being captured from a second point of view;
- processing the first image using an object recognition model to identify a plurality of first object regions in the first image, each of the first object regions corresponding to a respective one of the objects captured by the first image;
- generating a plurality of first object images based on the first object regions, each of the first object images being based on a respective one of the first object regions and capturing a respective one of the objects;
- processing the second image using the object recognition model to identify a plurality of second object regions in the second image, each of the second object regions corresponding to a respective one of the objects captured by the second image;
- generating a plurality of second object images based on the second object regions, each of the second object images being based on a respective one of the second object regions and capturing a respective one of the objects;
- processing each of the first object images using an object-contrastive model to generate respective first object image embeddings, wherein the object-contrastive model is a deep neural network model;
- processing each of the second object images using the object-contrastive model to generate respective second object image embeddings;
- generating a plurality of anchor, positive pairs that each include a respective one of the first object image embeddings and a respective one of second object image embeddings determined to be most similar to the respective one of the first object image embeddings;
- generating at least one metric loss based on contrasting the anchor, positive pairs with anchor, negative pairs that each include a respective one of the first object image embeddings and a respective one of the second object image embeddings not determined to be most similar to the respective one of the first object image embeddings;
- updating the object-contrastive model based on the at least one metric loss.
8 more elements
A system comprising instructions stored in memory, and one or more processors executing the stored instructions to perform a method that includes:
- identifying a first image of an environment, the first image capturing objects in the environment and being captured from a first point of view;
- identifying a second image of the environment, the second image capturing at least a subset of the objects in the environment and being captured from a second point of view;
- processing the first image using an object recognition model to identify a plurality of first object regions in the first image, each of the first object regions corresponding to a respective one of the objects captured by the first image;
- generating a plurality of first object images based on the first object regions, each of the first object images being based on a respective one of the first object regions and capturing a respective one of the objects;
- processing the second image using the object recognition model to identify a plurality of second object regions in the second image, each of the second object regions corresponding to a respective one of the objects captured by the second image;
- generating a plurality of second object images based on the second object regions, each of the second object images being based on a respective one of the second object regions and capturing a respective one of the objects;
- processing each of the first object images using an object-contrastive model to generate respective first object image embeddings, wherein the object-contrastive model is a deep neural network model;
- processing each of the second object images using the object-contrastive model to generate respective second object image embeddings;
- generating a plurality of anchor, positive pairs that each include a respective one of the first object image embeddings and a respective one of second object image embeddings determined to be most similar to the respective one of the first object image embeddings;
- generating at least one metric loss based on contrasting the anchor, positive pairs with anchor, negative pairs that each include a respective one of the first object image embeddings and a respective one of the second object image embeddings not determined to be most similar to the respective one of the first object image embeddings;
- updating the object-contrastive model based on the at least one metric loss.
8 more elements
Dependencies are inferred from the claim text.