# US 2024/0201832 A1 · Generating masked regions of an image using a predicted user intent

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0201832 A1
- Document ID: US20240201832A1
- Status: Published application
- Kind: A1
- Application number: 18/066,199
- Filed: 2022-12-14
- Published: 2024-06-20
- Earliest claimed: 2022-12-14
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Brian PRICE; Yifei FAN; Joshua MYERS-DEAN
- Web page: https://patentgrep.com/patent/US20240201832A1

## CPC

- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space
- G06V 10/82: using neural networks
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments are disclosed for predicting a user's intent to generate a masked region of an image based on a user gesture. The method may include receiving a user interaction corresponding to a region in an image. The user interaction is used to select the object in the image. The method further includes generating a feature map using a first machine learning model and the user interaction, where the feature map includes at least one feature corresponding to a mode type. The mode type is an intended operation of a user. Lastly, the method includes generating a mask of the object in the image using a second machine learning model, wherein the mask is generated responsive to the mode type.

## Claims

### Claim 1

A method comprising:
receiving a user interaction corresponding to a region in an image;
generating a feature map using a first machine learning model and the user interaction, wherein the feature map includes a feature corresponding to a mode type; and
generating a mask of the region in the image using a second machine learning model, wherein the mask is generated responsive to the mode type.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the feature map includes a feature corresponding to a gesture type, and wherein
generating the mask of the region in the image using the second machine learning model further includes generating the mask of the region responsive to the gesture type.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving an initial mask of the region in the image; and
generating the mask of the region in the image using the second machine learning model, wherein the mask is generated responsive to the mode type and the initial mask.

### Claim 4 (depends on 1)

The method of claim 1, wherein the second machine learning model is an ensemble of gesture-specific decoders including:
an addition decoder configured to generate a first mask of the region in the image by adding pixels associated with the user interaction;
a removal decoder configured to generate a second mask of the region in the image by removing pixels associated with the user interaction; and
a mask selector configured to select the generated mask based on the first mask, the second mask, and the mode type.

### Claim 5 (depends on 1)

The method of claim 1, wherein the second machine learning module is an ensemble of mode-specific decoders including a gesture decoder corresponding to each gesture of a plurality of gestures.

### Claim 6 (depends on 5)

The method of claim 5, wherein the plurality of gestures includes a click, a lasso, a bounding box, and a stroke.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the mode type is an add mode, and wherein
generating the mask of the region in the image includes adding one or more pixels corresponding to one or more pixels of the region.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the mode type is a remove mode, and wherein
generating the mask of the region in the image includes removing one or more pixels corresponding to one or more pixels of the region.

### Claim 9

A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
receiving a user interaction corresponding to a region in an image;
generating a feature vector using an encoder and the user interaction;
generating a first classification using a first classifier and the feature vector;
generating, a mask of the region using a decoder, the feature vector and the first classification; and
outputting a masked region based on the generated mask.

### Claim 10 (depends on 9)

The system of claim 9, wherein generating the first classification using the first classifier and the feature vector comprises:
generating a mode type using a mode classifier, or
generating a gesture type using a gesture classifier.

### Claim 11 (depends on 9)

The system of claim 9, wherein the processing device performs further operations comprising:
generating a second classification using a second classifier and the feature vector; and
generating the mask of the region using the decoder, the feature vector, the first classification, and the second classification.

### Claim 12 (depends on 9)

The system of claim 9, wherein the decoder includes at least one of:
an addition decoder configured to generate a first mask of the region in the image by adding pixels associated with the user interaction; or
a removal decoder configured to generate a second mask of the region in the image by removing pixels associated with the user interaction.

### Claim 13 (depends on 12)

The system of claim 12, wherein the processing device performs further operations comprising:
selecting the generated mask based on the first mask, the second mask, and a mode type.

### Claim 14 (depends on 9)

The system of claim 9, wherein the decoder is a gesture decoder of a plurality of gesture decoders, wherein the plurality of gesture decoders each correspond to a different gesture of a plurality of gestures.

### Claim 15 (depends on 14)

The system of claim 14, wherein the plurality of gestures include a click, a lasso, a bounding box, and a stroke.

### Claim 16 (depends on 9)

The system of claim 9, wherein:
the decoder is a gesture decoder of a plurality of gesture decoders,
the first classifier is a gesture classifier and the first classification is a gesture type, and
the processing device performs further operations comprising:
triggering the gesture decoder corresponding to a gesture type.

### Claim 17

A computer-readable medium storing instructions that, when executed by a processor, cause the processor to:
receive a set of inputs including an image and at least one of a gesture map corresponding to a user interaction with a region in the image, or a distance map corresponding to a distance of the user interaction from the region in the image;
generate a feature vector using an encoder based on the set of inputs;
determine a mode type using the feature vector and a first classifier;
determine a gesture type using the feature vector and a second classifier;
generate, using a decoder, a mask of the region, based on the feature vector, and at least one of the mode type or the gesture type; and
output a masked region based on the generated mask.

### Claim 18 (depends on 17)

The computer-readable medium of claim 17, wherein the decoder is one decoder of a plurality of decoders, and the plurality of decoders include:
an addition decoder configured to generate a first mask of the region in the image by adding pixels associated with the user interaction; and
a removal decoder configured to generate a second mask of the region in the image by removing pixels associated with the user interaction.

### Claim 19 (depends on 18)

The computer-readable medium of claim 18, storing instructions that further cause the processor to:
select the generated mask based on the first mask, the second mask, and the mode type.

### Claim 20

The computer-readable medium of 17, wherein the decoder is one decoder of a plurality of decoders, and the decoder of the plurality of decoders is triggered based on the gesture type, wherein the plurality of decoders includes a click decoder, a lasso decoder, a bounding box decoder, and a stroke decoder.