Open vocabulary instance segmentation with noise estimation and robust student

A method for training a machine learning model, comprising:

• receiving a training image and a caption for the training image, wherein the caption includes text describing an object in the training image;
• generating a plurality of candidate regions for the object using a teacher network;
• generating features corresponding to each of the plurality of candidate regions;
• generating text features for the caption using a text encoder;
• comparing the features corresponding to each of the plurality of candidate regions to the text features by mapping the features corresponding to each of the plurality of candidate regions into an embedding space of the text features;
• generating, using the teacher network, a pseudo mask for the object based on the text describing the object and the comparison;
• generating a mask for the object using a student network;
• and
• updating parameters of the student network based on the mask, the pseudo mask, and the text features.

An apparatus comprising:

• at least one processor;
• at least one memory including instructions executable by the at least one processor;
• and
• a machine learning model comprising parameters in the at least one memory and configured to generate a mask for an element in an image, wherein the machine learning model is trained using steps comprising:
• receiving an annotated training set, a training image and a caption for the training image, wherein each image in the annotated training set is associated with ground truth annotations, and wherein the caption includes text describing an object in the training image;
• generating predicted annotations for each image in the annotated training set using a student network;
• computing a ground truth loss by comparing the predicted annotations to the ground truth annotations;
• generating a pseudo mask for the object using a teacher network based on the text describing the object;
• generating a mask for the object using the student network separate from the teacher network;
• estimating, using a noise estimation network, a noise level for the pseudo mask generated by the teacher network;
• modifying the mask generated by the student network based on the noise level to obtain a noise-adjusted mask;
• and
• updating parameters of the student network based on the ground truth loss and the noise-adjusted mask.