# US 2025/0148752 A1 · Open vocabulary image segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148752 A1
- Document ID: US20250148752A1
- Status: Published application
- Kind: A1
- Application number: 18/502,719
- Filed: 2023-11-06
- Published: 2025-05-08
- Earliest claimed: 2023-11-06
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Vibashan VISHNUKUMAR SHARMINI; Shubhankar Mangesh BORSE; Hyojin PARK; Debasmit DAS; Munawar HAYAT; Fatih Murat PORIKLI
- Web page: https://patentgrep.com/patent/US20250148752A1

## CPC

- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/82: using neural networks
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 30/148: Segmentation of character regions
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. In an example method, an input image is accessed, and the input image is processed using an image encoder to generate an image embedding tensor. The image embedding tensor is processed using a mask decoder machine learning model to generate a set of mask embedding tensors. A textual input is processed using a text encoder to generate a text embedding tensor. A set of augmented masks is generated based on aggregating the text embedding tensor with the set of mask embedding tensors.

## Claims

### Claim 1

A processing system comprising:
one or more memories comprising processor-executable instructions; and
one or more processors configured to execute the processor-executable instructions and cause the processing system to:
access an input image;
process the input image using an image encoder to generate an image embedding tensor;
process the image embedding tensor using a mask decoder machine learning model to generate a set of mask embedding tensors;
process a textual input using a text encoder to generate a text embedding tensor; and
generate a set of augmented masks based on aggregating the text embedding tensor with the set of mask embedding tensors.

### Claim 2 (depends on 1)

The processing system of claim 1, wherein, to generate the set of augmented masks, the one or more processors are configured to execute the processor-executable instructions and cause the processing system to:
generate a set of masks based on the set of mask embedding tensors and the image embedding tensor;
generate a set of predictions based on aggregating the text embedding tensor with the set of mask embedding tensors; and
associate the set of predictions with the set of masks.

### Claim 3 (depends on 2)

The processing system of claim 2, wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system to generate, based on a first augmented mask of the set of augmented masks, a classification indicating that a region of the image corresponding to the first augmented mask depicts an entity corresponding to the textual input.

### Claim 4 (depends on 3)

The processing system of claim 3, wherein:
the mask decoder was trained based at least in part on a set of category text embedding tensors, and
the set of category text embedding tensors does not include the text embedding.

### Claim 5 (depends on 1)

The processing system of claim 1, wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system to:
sample a first set of points within a first mask corresponding to a first augmented mask of the set of augmented masks; and
generate a first updated mask based on processing the image embedding tensor and the first set of points using a second decoder.

### Claim 6 (depends on 5)

The processing system of claim 5, wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system to:
sample a second set of points within the first updated mask; and
generate a second updated mask based on processing the image embedding tensor and the second set of points using the second decoder.

### Claim 7 (depends on 1)

The processing system of claim 1, wherein, to aggregate the text embedding tensor with the set of mask embedding tensors, the one or more processors are configured to execute the processor-executable instructions and cause the processing system to compute a dot product between the text embedding tensor and the set of mask embedding tensors.

### Claim 8 (depends on 1)

The processing system of claim 1, wherein the mask decoder was trained to perform panoptic segmentation based at least in part on:
a training image,
a set of training mask embedding tensors for the training image, and
a set of category text embedding tensors.

### Claim 9 (depends on 1)

The processing system of claim 1, wherein the image encoder comprises:
a component having parameters that were not trained based on semantic meaning of input images; and
one or more auxiliary parameters that were trained based on embeddings generated by a semantic image encoder.

### Claim 10

A processor-implemented method, comprising:
accessing an input image;
processing the input image using an image encoder to generate an image embedding tensor;
processing the image embedding tensor using a mask decoder machine learning model to generate a set of mask embedding tensors;
processing a textual input using a text encoder to generate a text embedding tensor; and
generating a set of augmented masks based on aggregating the text embedding tensor with the set of mask embedding tensors.

### Claim 11 (depends on 10)

The processor-implemented method of claim 10, wherein generating the set of augmented masks comprises:
generating a set of masks based on the set of mask embedding tensors and the image embedding tensor;
generating a set of predictions based on aggregating the text embedding tensor with the set of mask embedding tensors; and
associating the set of predictions with the set of masks.

### Claim 12 (depends on 11)

The processor-implemented method of claim 11, further comprising generating, based on a first augmented mask of the set of augmented masks, a classification indicating that a region of the image corresponding to the first augmented mask depicts an entity corresponding to the textual input.

### Claim 13 (depends on 12)

The processor-implemented method of claim 12, wherein:
the mask decoder machine learning model was trained based at least in part on a set of category text embedding tensors, and
the set of category text embedding tensors does not include the text embedding.

### Claim 14 (depends on 10)

The processor-implemented method of claim 10, further comprising:
sampling a first set of points within a first mask corresponding to a first augmented mask of the set of augmented masks; and
generating a first updated mask based on processing the image embedding tensor and the first set of points using a second decoder.

### Claim 15 (depends on 14)

The processor-implemented method of claim 14, further comprising:
sampling a second set of points within the first updated mask; and
generating a second updated mask based on processing the image embedding tensor and the second set of points using the second decoder.

### Claim 16 (depends on 10)

The processor-implemented method of claim 10, wherein aggregating the text embedding tensor with the set of mask embedding tensors comprises computing a dot product between the text embedding tensor and the set of mask embedding tensors.

### Claim 17 (depends on 10)

The processor-implemented method of claim 10, wherein the mask decoder machine learning model was trained to perform panoptic segmentation based at least in part on:
a training image,
a set of training mask embedding tensors for the training image, and
a set of category text embedding tensors.

### Claim 18 (depends on 10)

The processor-implemented method of claim 10, wherein the image encoder comprises:
a component having parameters that were not trained based on semantic meaning of input images; and
one or more auxiliary parameters that were trained based on embeddings generated by a semantic image encoder.

### Claim 19

One or more non-transitory computer-readable media comprising processor-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to:
access an input image;
process the input image using an image encoder to generate an image embedding tensor;
process the image embedding tensor using a mask decoder machine learning model to generate a set of mask embedding tensors;
process a textual input using a text encoder to generate a text embedding tensor; and
generate a set of augmented masks based on aggregating the text embedding tensor with the set of mask embedding tensors.

### Claim 20 (depends on 19)

The non-transitory computer-readable media of claim 19, wherein, to generate the set of augmented masks, the instructions cause the processing system to:
generate a set of masks based on the set of mask embedding tensors and the image embedding tensor;
generate a set of predictions based on aggregating the text embedding tensor with the set of mask embedding tensors; and
associate the set of predictions with the set of masks.

### Claim 21 (depends on 20)

The non-transitory computer-readable media of claim 20, wherein the instructions further cause the processing system to generate, based on a first augmented mask of the set of augmented masks, a classification indicating that a region of the image corresponding to the first augmented mask depicts an entity corresponding to the textual input.

### Claim 22 (depends on 21)

The non-transitory computer-readable media of claim 21, wherein:
the mask decoder machine learning model was trained based at least in part on a set of category text embedding tensors, and
the set of category text embedding tensors does not include the text embedding.

### Claim 23 (depends on 19)

The non-transitory computer-readable media of claim 19, wherein the instructions further cause the processing system to:
sample a first set of points within a first mask corresponding to a first augmented mask of the set of augmented masks; and
generate a first updated mask based on processing the image embedding tensor and the first set of points using a second decoder.

### Claim 24 (depends on 23)

The non-transitory computer-readable media of claim 23, wherein the instructions further cause the processing system to:
sample a second set of points within the first updated mask; and
generate a second updated mask based on processing the image embedding tensor and the second set of points using the second decoder.

### Claim 25 (depends on 19)

The non-transitory computer-readable media of claim 19, wherein, to aggregate the text embedding tensor with the set of mask embedding tensors, the instructions cause the processing system to compute a dot product between the text embedding tensor and the set of mask embedding tensors.

### Claim 26 (depends on 19)

The non-transitory computer-readable media of claim 19, wherein the mask decoder machine learning model was trained to perform panoptic segmentation based at least in part on:
a training image,
a set of training mask embedding tensors for the training image, and
a set of category text embedding tensors.

### Claim 27 (depends on 19)

The non-transitory computer-readable media of claim 19, wherein the image encoder comprises:
a component having parameters that were not trained based on semantic meaning of input images; and
one or more auxiliary parameters that were trained based on embeddings generated by a semantic image encoder.

### Claim 28

A processing system, comprising:
means for accessing an input image;
means for processing the input image using an image encoder to generate an image embedding tensor;
means for processing the image embedding tensor using a mask decoder machine learning model to generate a set of mask embedding tensors;
means for processing a textual input using a text encoder to generate a text embedding tensor; and
means for generating a set of augmented masks based on aggregating the text embedding tensor with the set of mask embedding tensors.

### Claim 29 (depends on 28)

The processing system of claim 28, wherein the means for generating the set of augmented masks comprise:
means for generating a set of masks based on the set of mask embedding tensors and the image embedding tensor;
means for generating a set of predictions based on aggregating the text embedding tensor with the set of mask embedding tensors; and
means for associating the set of predictions with the set of masks.

### Claim 30 (depends on 28)

The processing system of claim 28, further comprising:
means for sampling a set of points within a first mask corresponding to an augmented mask of the set of augmented masks; and
means for generating an updated mask based on processing the image embedding tensor and the set of points using a second decoder.
