# US 2026/0094404 A1 · Segmentation of objects in an image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0094404 A1
- Document ID: US20260094404A1
- Status: Published application
- Kind: A1
- Application number: 19/113,362
- Filed: 2024-05-09
- Published: 2026-04-02
- Earliest claimed: 2023-05-09
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Bryan FELDMAN
- Web page: https://patentgrep.com/patent/US20260094404A1

## CPC

- G06V 10/82: using neural networks
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 20/38: {Outdoor scenes}
- G06V 20/50: Context or environment of the image
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30201: Face
- H04M 1/724: User interfaces specially adapted for cordless or mobile telephones
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

A media application performs object recognition on an initial image to identify a set of objects in the initial image. The media application determines whether the initial image is an outdoor scene. Responsive to the initial image being an outdoor scene, the media application determining a sky segment from the initial image. The media application determines whether the initial image includes a subject that is human or animal. Responsive to the initial image including the subject, the media application determines a subject segment from the initial image. The media application receives at a user interface that includes the initial image, user input corresponding to selection of a selected object from the set of objects. The media application updates the user interface to include an indication that the selected object was selected.

## Claims

### Claim 1

A computer-implemented method comprising:
performing object recognition on an initial image to identify a set of objects in the initial image;
determining whether the initial image is an outdoor scene;
responsive to the initial image being an outdoor scene, determining a sky segment from the initial image;
determining whether the initial image includes a subject that is human or animal;
responsive to the initial image including the subject, determining a subject segment from the initial image;
determining whether the initial image includes one or more distracting objects;
responsive to the initial image including one or more distracting objects, determining one or more distracting segments from the initial image;
receiving, at a user interface that includes the initial image, user input corresponding to selection of a selected object from the set of objects; and
updating the user interface to include an indication that the selected object was selected.

### Claim 2 (depends on 1)

The method of claim 1, wherein the user input includes multiple taps of the selected object and further comprising:
determining a number of taps from the user input; and
determining the selected object based on the number of taps, wherein a first tap is associated with a different region than a second tap.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
responsive to the initial image including the subject, generating a background segment, wherein the subject segment is associated with a foreground region, the background segment is associated with a background region, and pixels in the initial image are associated with the foreground region or the background region; and
determining that the user input corresponds to the foreground region based on the user input making contact with pixels that are associated with the foreground region.

### Claim 4 (depends on 1)

The method of claim 1, wherein performing object recognition to identify objects in the initial image includes determining object bounding boxes for each of the objects and the method further comprises:
determining that the user input corresponds to the selected object based on a proximity of the user input to a closest object bounding box.

### Claim 5 (depends on 1)

The method of claim 1, wherein a convolutional neural network (CNN) performs segmentation and the method further comprises:
providing the initial image and a heatmap of keypoints as input to the CNN; and
outputting, with the convolutional neural network, segmentation masks that correspond to the sky segment, the subject segment, and the one or more distracting segments.

### Claim 6 (depends on 1)

The method of claim 1, wherein the user input includes selection of a sky and the method further comprises:
receiving a request from a user to change a lighting in the initial image;
providing, as input to a diffusion model, an initial image and a request to change a lighting in the initial image; and
outputting, with the diffusion model, an output image that satisfies the request.

### Claim 7 (depends on 1)

The method of claim 1, wherein the user input includes selection of one or more background objects for removal and the method further comprises:
removing the one or more distracting objects from the initial image based on object recognition; and
generating a modified image that includes inpainting of pixels associated with the one or more distracting objects.

### Claim 8 (depends on 1)

The method of claim 1, wherein the selected object is an incomplete object and an omitted portion of the incomplete object is cut off by a boundary of the initial image or obscured by another object, the method further comprising:
generating a preserving mask that includes the incomplete object;
removing the incomplete object from the initial image;
generating an inpainted image that replaces incomplete object pixels corresponding to the incomplete object with background pixels that match a background in the initial image;
providing, as input to a diffusion model, the preserving mask, the incomplete object, and the inpainted image;
outputting, with the diffusion model, a complete object; and
generating a modified image by blending one or more versions of the complete object with one or more versions of the inpainted image using the preserving mask.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
receiving a textual request to change the selected object in the initial image;
determining, from the initial image, a face segment for a face of the subject based on the subject segment;
generating a preserving mask that corresponds to the face segment;
providing the textual request, the initial image, and the preserving mask as input to a diffusion model; and
outputting, with the diffusion model, an output image that satisfies the textual request.

### Claim 10

A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
performing object recognition on an initial image to identify a set of objects in the initial image;
determining whether the initial image is an outdoor scene;
responsive to the initial image being an outdoor scene, determining a sky segment from the initial image;
determining whether the initial image includes a subject that is human or animal;
responsive to the initial image including the subject, determining a subject segment from the initial image;
determining whether the initial image includes one or more distracting objects;
responsive to the initial image including one or more distracting objects, determining one or more distracting segments from the initial image;
receiving, at a user interface that includes the initial image, user input corresponding to selection of a selected object from the set of objects; and
updating the user interface to include an indication that the selected object was selected.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the user input includes multiple taps of the selected object and the operations further include:
determining a number of taps from the user input; and
determining the selected object based on the number of taps, wherein a first tap is associated with a different region than a second tap.

### Claim 12 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the operations further include:
responsive to the initial image including the subject, generating a background segment, wherein the subject segment is associated with a foreground region, the background segment is associated with a background region, and pixels in the initial image are associated with the foreground region or the background region; and
determining that the user input corresponds to the foreground region based on the user input making contact with pixels that are associated with the foreground region.

### Claim 13 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein performing object recognition to identify objects in the initial image includes determining object bounding boxes for each of the objects and the operations further include:
determining that the user input corresponds to the selected object based on a proximity of the user input to a closest object bounding box.

### Claim 14 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein a convolutional neural network (CNN) performs segmentation and the operations further include:
providing the initial image and a heatmap of keypoints as input to the CNN; and
outputting, with the convolutional neural network, segmentation masks that correspond to the sky segment, the subject segment, and the one or more distracting segments.

### Claim 15 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the user input includes selection of a sky and the operations further include:
receiving a request from a user to change a lighting in the initial image;
providing, as input to a diffusion model, an initial image and a request to change a lighting in the initial image; and
outputting, with the diffusion model, an output image that satisfies the request.

### Claim 16

A system comprising:
a processor; and
a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
performing object recognition on an initial image to identify a set of objects in the initial image;
determining whether the initial image is an outdoor scene;
responsive to the initial image being an outdoor scene, determining a sky segment from the initial image;
determining whether the initial image includes a subject that is human or animal;
responsive to the initial image including the subject, determining a subject segment from the initial image;
determining whether the initial image includes one or more distracting objects;
responsive to the initial image including one or more distracting objects, determining one or more distracting segments from the initial image;
receiving, at a user interface that includes the initial image, user input corresponding to selection of a selected object from the set of objects; and
updating the user interface to include an indication that the selected object was selected.

### Claim 17 (depends on 16)

The system of claim 16, wherein the user input includes multiple taps of the selected object and the operations further include:
determining a number of taps from the user input; and
determining the selected object based on the number of taps, wherein a first tap is associated with a different region than a second tap.

### Claim 18 (depends on 16)

The system of claim 16, wherein the operations further include:
responsive to the initial image including the subject, generating a background segment, wherein the subject segment is associated with a foreground region, the background segment is associated with a background region, and pixels in the initial image are associated with the foreground region or the background region; and
determining that the user input corresponds to the foreground region based on the user input making contact with pixels that are associated with the foreground region.

### Claim 19 (depends on 16)

The system of claim 16, wherein performing object recognition to identify objects in the initial image includes determining object bounding boxes for each of the objects and the operations further include:
determining that the user input corresponds to the selected object based on a proximity of the user input to a closest object bounding box.

### Claim 20 (depends on 16)

The system of claim 16, wherein a convolutional neural network (CNN) performs segmentation and the operations further include:
providing the initial image and a heatmap of keypoints as input to the CNN; and
outputting, with the convolutional neural network, segmentation masks that correspond to the sky segment, the subject segment, and the one or more distracting segments.
