System and method for model-free, one-shot object pose estimation via coordinate regression

A method using an object-agnostic model, implemented using an electronic processor and memory, for predicting a pose of an object in an image, comprising:

• (a) receiving a query image having a target object therein;
• (b) receiving a set of reference images of the target object from different viewpoints;
• (c) encoding, using a vision transformer, the received query image and the received set of reference images to generate a set of token features for the received query image and a set of token features for the received set of reference images;
• (d) extracting, using a transformer decoder, information from the set of token features for the encoded reference images with respect to a set of token features for the received query image;
• (e) processing, using a prediction head, the combined set of token features to generate a 2D-3D mapping of the query image;
• and
• (f) processing the 2D-3D mapping to determine the pose of the target object in the query image.

A computer-implemented method of training a machine learning model for regression on pixel-level annotations in images, the machine learning model comprising an image encoder, a feature mixer, and a decoder, the method comprising:

• pre-training the image encoder and the decoder for cross-view completion on images;
• constructing training tuples, each training tuple comprising a first image and one or more second images, wherein the first image is associated with dense pixel-level annotations, and wherein each of the one or more second images is associated with sparse pixel-level annotations;
• generating, by the image encoder, a set of first image tokens from the first image;
• generating, by the image encoder, one or more sets of second image tokens from the one or more second images;
• generating, by the feature mixer, one or more sets of augmented second image tokens by augmenting each set of second image tokens with encodings of the respective sparse pixel-level annotations associated with the respective set of second image tokens, wherein the augmenting includes mixing the respective set of second image tokens and the encodings of the respective sparse pixel-level annotations;
• processing, by the decoder, the set of first image tokens and the one or more sets of augmented second image tokens to generate prediction data of the machine learning model for the first image, wherein the processing comprises receiving a set of augmented second image tokens of the one or more sets of augmented second image tokens, wherein the prediction data comprises predictions for each image pixel of the first image and confidences for the predictions;
• and
• fine-tuning the machine learning model, wherein the fine-tuning comprises adjusting parameters of the image encoder, the feature mixer, and the decoder to minimize a loss function, wherein the loss function is based on the prediction data and the dense pixel-level annotations.