Semantic labeling of images with generative language model

A method for image processing, the method comprising:

• receiving an image;
• computing a segmentation mask that identifies a region of interest included in the image;
• computing a plurality of encoded image features based at least in part on the image;
• receiving a text instruction that includes a plurality of instruction text tokens;
• at a visual resampler that includes a plurality of transformer layers, computing a mask query based at least in part on the segmentation mask, the plurality of encoded image features, and the text instruction, the mask query including a plurality of mask query text tokens that are iteratively recomputed over the plurality of transformer layers;
• receiving a natural language query that includes the mask query and the text instruction;
• and
• based at least in part on the natural language query, generating a semantic label associated with the region of interest;
• and
• outputting the semantic label;
• and
• further comprising:
• receiving a mode query that indicates a vocabulary specificity mode;
• and
• computing the mask query based at least in part on the mode query.

A computing system comprising:

• one or more processing devices configured to:
• receive an image;
• compute a segmentation mask that identifies a region of interest included in the image;
• compute a plurality of encoded image features based at least in part on the image;
• compute a context query associated with a bounding box that surrounds the region of interest;
• receive a text instruction that includes a plurality of instruction text tokens;
• receive a mode query that indicates a vocabulary specificity mode;
• at a visual resampler that includes a plurality of transformer layers, compute a mask query based at least in part on the segmentation mask, the plurality of encoded image features, the context query, the text instruction, and the mode query, the mask query including a plurality of mask query text tokens that are iteratively recomputed over the plurality of transformer layers;
• receive a natural language query that includes the mask query and the text instruction;
• based at least in part on the natural language query, generate a semantic label associated with the region of interest;
• and
• output the semantic label.