# US 12,591,950 B2 · Iteratively applying neural networks to automatically segment objects portrayed in digital images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,591,950 B2
- Document ID: US12591950B2
- Status: Granted
- Kind: B2
- Application number: 17/584,170
- Filed: 2022-01-25
- Published: 2022-05-12
- Granted: 2026-03-31
- Earliest claimed: 2018-05-01
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: I-Ming Pao; Zhe Lin; Sarah Stuckey; Jianming Zhang; Betty Leong
- Web page: https://patentgrep.com/patent/US12591950B2

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 10/82: using neural networks
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure relates to systems, method, and computer readable media that iteratively apply a neural network to a digital image at a reduced resolution to automatically identify pixels of salient objects portrayed within the digital image. For example, the disclosed systems can generate a reduced-resolution digital image from an input digital image and apply a neural network to identify a region corresponding to a salient object. The disclosed systems can then iteratively apply the neural network to additional reduced-resolution digital images (based on the identified region) to generate one or more reduced-resolution segmentation maps that roughly indicate pixels of the salient object. In addition, the systems described herein can perform post-processing based on the reduced-resolution segmentation map(s) and the input digital image to accurately determine pixels that correspond to the salient object.

## Claims

### Claim 1

A system comprising:
one or more memory devices comprising a neural network; and
at least one processor configured to cause the system to:
determine a region of a digital image having a first resolution and comprising a single salient object;
generate, from the region of the digital image comprising the single salient object, a segmentation map of the single salient object having a second resolution corresponding to a predetermined image resolution specification of the neural network by utilizing the neural network to identify pixels of the digital image corresponding to the single salient object in the region of the digital image;
determine whether to perform additional refinement to the segmentation map of the region of the digital image comprising the single salient object;
generate, from a sub-region of the region of the digital image and based on determining to perform additional refinement to the segmentation map, an additional segmentation map of the sub-region of the region having the second resolution by utilizing the neural network to refine identification of the pixels corresponding to the single salient object in the sub-region; and
generate, from the digital image, the segmentation map of the region of the digital image, and the additional segmentation map of the sub-region, a refined segmentation map of the digital image having the first resolution that identifies the pixels in the digital image corresponding to the single salient object.

### Claim 2 (depends on 1)

The system of claim 1, wherein the at least one processor is further configured cause the system to generate the refined segmentation map of the digital image without user input indicating a portion of the digital image corresponding to the single salient object.

### Claim 3 (depends on 1)

The system of claim 1, wherein the at least one processor is further configured cause the system to generate a modified digital image utilizing the refined segmentation map.

### Claim 4 (depends on 1)

The system of claim 1, wherein the at least one processor is further configured cause the system to generate a version of the digital image having a reduced-resolution and generate the additional segmentation map and the segmentation map of the region of the digital image comprising the single salient object from the version of the digital image having the reduced-resolution.

### Claim 5 (depends on 4)

The system of claim 4, wherein the at least one processor is further configured cause the system to upsample the refined segmentation map from the reduced-resolution to a higher resolution.

### Claim 6 (depends on 5)

The system of claim 5, wherein the at least one processor is further configured to cause the system to upsample the refined segmentation map from the reduced-resolution to the higher resolution by applying at least one of a dense conditional random field filter, a guided filter, or a graph cut filter to the refined segmentation map of the sub-region of the region of the digital image.

### Claim 7 (depends on 1)

The system of claim 1, wherein utilizing the neural network to refine identification of the pixels corresponding to the single salient object in the sub-region comprises:
determining confidence values corresponding to the pixels;
determining that one or more confidence values corresponding to one or more pixels that do not satisfy a threshold condition; and
refining the one or more confidence values utilizing the neural network.

### Claim 8 (depends on 1)

The system of claim 1, wherein the neural network comprises a convolution neural network trained to identify salient objects in digital images.

### Claim 9

A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to perform operations comprising:
generating, utilizing a neural network, an initial segmentation map of a digital image that identifies pixels of the digital image corresponding to a single salient object portrayed in the digital image, the digital image having a first resolution;
generating, from a region of the digital image comprising the single salient object as indicated in the initial segmentation map, a segmentation map of the single salient object portrayed utilizing the neural network to identify pixels of the digital image corresponding to the single salient object in the region of the digital image, the initial segmentation map having a second resolution corresponding to a predetermined image resolution specification of the neural network;
determining whether to perform additional refinement to the segmentation map of the region of the digital image;
generating, from a sub-region of the region of the digital image and based on determining to perform additional refinement to the segmentation map, an additional segmentation map of the sub-region of the region having the second resolution by utilizing the neural network to refine identification of the pixels corresponding to the single salient object in the sub-region;
identifying, based on the additional segmentation map, one or more regions of the digital image with confidence values meeting a threshold condition; and
generating a refined segmentation map having the first resolution by reprocessing, utilizing the neural network, the one or more regions of the digital image to refine the confidence values meeting the threshold condition.

### Claim 10 (depends on 9)

The non-transitory computer readable medium of claim 9, wherein generating the refined segmentation map by reprocessing, utilizing the neural network, the one or more regions of the digital image to refine the confidence values meeting the threshold condition comprises iteratively applying the neural network to regions of the digital image with confidence values meeting the threshold condition as indicated by a prior application of the neural network.

### Claim 11 (depends on 10)

The non-transitory computer readable medium of claim 10, wherein the threshold condition comprises one or more of:
a predefined number of iterations of applying the neural network;
a predefined duration of time; or
a threshold convergence of confidence values between consecutive iterations.

### Claim 12 (depends on 9)

The non-transitory computer readable medium of claim 9, further comprising instructions that, when executed by the at least one processor, cause the computing device to perform operations comprising upsampling the refined segmentation map from a reduced-resolution to a higher resolution.

### Claim 13 (depends on 12)

The non-transitory computer readable medium of claim 12, wherein upsampling the refined segmentation map from the reduced-resolution to the higher resolution comprises applying at least one of a dense conditional random field filter, a guided filter, or a graph cut filter to the refined segmentation map.

### Claim 14 (depends on 12)

The non-transitory computer readable medium of claim 12, wherein upsampling the refined segmentation map from the reduced-resolution to the higher resolution comprises upsampling based on information from a version of the digital image having the higher resolution.

### Claim 15

In a digital medium environment for editing high-resolution digital images, a computer-implemented method comprising:
receiving user input selecting an option to perform object selection on a digital image;
generating, utilizing a neural network, an initial segmentation map of the digital image that identifies pixels of the digital image corresponding to a single salient object portrayed in the digital image, the digital image having a first resolution;
generating, from a region of the digital image comprising the single salient object as indicated in the initial segmentation map, a segmentation map of the single salient object having a second resolution corresponding to a predetermined image resolution specification of the neural network by utilizing the neural network to identify pixels of the region of the digital image corresponding to the single salient object without receiving user input indicating a portion of the digital image corresponding to the single salient object;
determining whether to perform additional refinement to the segmentation map of the region of the digital image;
generating, from a sub-region of the region of the digital image and based on determining to perform additional refinement to the segmentation map, an additional segmentation map of the sub-region having the second resolution by utilizing the neural network to refine identification of the pixels corresponding to the single salient object in the sub-region; and
upsampling the additional segmentation map from the second resolution to the first resolution.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, wherein determining whether to perform additional refinement to the segmentation map of the region of the digital image comprises:
analyzing the segmentation map of the single salient object to identify the sub-region of the digital image comprising pixels having confidence values, indicating whether the pixels belong to the single salient object or not, below a threshold value.

### Claim 17 (depends on 15)

The computer-implemented method of claim 15, further comprising:
generating a version of the digital image with the second resolution by downsampling the digital image; and
utilizing the version of the digital image with the second resolution to generate the segmentation map and the additional segmentation map.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein upsampling the additional segmentation map from the second resolution to the first resolution comprises upsampling the additional segmentation map to a resolution of the digital image prior to the downsampling of the digital image.

### Claim 19 (depends on 15)

The computer-implemented method of claim 15, wherein upsampling the additional segmentation map from the second resolution to the first resolution comprises applying at least one of a dense conditional random field filter, a guided filter, or a graph cut filter to the additional segmentation map.

### Claim 20 (depends on 15)

The computer-implemented method of claim 15, wherein generating, from the digital image, the segmentation map of the portion of the digital image comprising the single salient object portrayed within the digital image by utilizing the neural network to identify pixels of the digital image corresponding to the single salient object comprises iteratively applying the neural network to regions of the digital image with confidence values meeting a threshold condition as indicated by a prior application of the neural network.