# US 2022/0044365 A1 · Automatically generating a trimap segmentation for a digital image by utilizing a trimap generation neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044365 A1
- Document ID: US20220044365A1
- Status: Published application
- Kind: A1
- Application number: 16/988,036
- Filed: 2020-08-07
- Published: 2022-02-10
- Earliest claimed: 2020-08-07
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: He Zhang; Seyed Morteza Safdarnejad; Yilin Wang; Zijun Wei; Jianming Zhang; Salil Tambe; Brian Price
- Web page: https://patentgrep.com/patent/US20220044365A1

## CPC

- G06T 7/194: involving foreground-background segmentation
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 7/11: Region-based segmentation
- G06T 2207/10024: Color image
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20132: Image cropping
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing a plurality of neural networks in a multi-branch pipeline to generate image masks for digital images. Specifically, the disclosed system can classify a digital image as a portrait or a non-portrait image. Based on classifying a portrait image, the disclosed system can utilize separate neural networks to generate a first mask portion for a portion of the digital image including a defined boundary region and a second mask portion for a portion of the digital image including a blended boundary region. The disclosed system can generate the mask portion for the blended boundary region by utilizing a trimap generation neural network to automatically generate a trimap segmentation including the blended boundary region. The disclosed system can then merge the first mask portion and the second mask portion to generate an image mask for the digital image.

## Claims

### Claim 1

A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor, cause a computing device to:
extract an estimated foreground region and an estimated background region of a digital image, wherein the estimated foreground region portrays an object in the digital image;
determine, utilizing a neural network, a blended boundary region of the object in the digital image, wherein the blended boundary region comprises a combination of foreground elements and background elements;
determine a trimap segmentation of the digital image by combining the estimated foreground region, the estimated background region, and the blended boundary region; and
generate, utilizing the trimap segmentation of the digital image, an image mask for the digital image.

### Claim 2 (depends on 1)

The non-transitory computer readable storage medium as recited in claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to:
utilize a semantic cropping neural network to crop the digital image to a portion of the digital image including the object;
resize the cropped digital image to a first resolution size; and
determine, utilizing the neural network, the blended boundary region of the object from the cropped digital image at the first resolution size.

### Claim 3 (depends on 2)

The non-transitory computer readable storage medium as recited in claim 2, further comprising instructions that, when executed by the at least one processor, cause the computing device to:
extract the estimated foreground region and the estimated background region from the cropped digital image at the first resolution size; and
determine the trimap segmentation of the digital image by combining the estimated foreground region, the estimated background region, and the blended boundary region at the first resolution size.

### Claim 4 (depends on 3)

The non-transitory computer readable storage medium as recited in claim 3, further comprising instructions that, when executed by the at least one processor, cause the computing device to:
upsample the cropped digital image and the trimap segmentation to a second resolution size; and
utilize a matting algorithm to generate the image mask from the upsampled cropped digital image and the upsampled trimap segmentation at the second resolution size.

### Claim 5 (depends on 4)

The non-transitory computer readable storage medium as recited in claim 4, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the image mask from the upsampled cropped digital image and the upsampled trimap segmentation at the second resolution size by:
utilizing the matting algorithm to generate an initial image mask at the second resolution size; and
upsampling the initial image mask to an original resolution of the digital image.

### Claim 6 (depends on 5)

The non-transitory computer readable storage medium as recited in claim 5, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the image mask by utilizing a guided image filter to refine a boundary between an updated foreground region and an updated background region in the initial image mask.

### Claim 7 (depends on 5)

The non-transitory computer readable storage medium as recited in claim 5, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the image mask by:
generating an updated foreground region and an updated background region by assigning, utilizing the matting algorithm, alpha values to portions of the blended boundary region based on the trimap segmentation; and
generating the image mask based on the updated foreground region and the updated background region according to the assigned alpha values.

### Claim 8 (depends on 2)

The non-transitory computer readable storage medium as recited in claim 2, further comprising instructions that, when executed by the at least one processor, cause the computing device to combine the image mask with an additional image mask corresponding to an additional portion of the digital image based on an original position of the cropped digital image relative to the digital image.

### Claim 9

A system comprising:
at least one computer memory device comprising a digital image; and
one or more servers configured to cause the system to:
determine an estimated foreground region and an estimated background region of the digital image, wherein the estimated foreground region portrays an object in the digital image;
determine, utilizing a neural network, a blended boundary region of the object in the digital image, wherein the blended boundary region comprises a combination of foreground elements and background elements;
determine a trimap segmentation of the digital image by combining the estimated foreground region, the estimated background region, and the blended boundary region;
generate an updated foreground region and an updated background region by utilizing a matting algorithm to assign an alpha value for each pixel within the blended boundary region of the digital image based on the trimap segmentation; and
generate an image mask for the digital image based on the updated foreground region and the updated background region according to the assigned alpha values.

### Claim 10 (depends on 9)

The system as recited in claim 9, wherein the one or more servers are further configured to cause the system to determine the estimated foreground region and the estimated background region of the digital image by:
detecting, utilizing a semantic cropping neural network, an object within a portion of the digital image;
cropping the digital image to the portion of the digital image including the object;
resizing the cropped digital image to a first resolution size; and
determining the estimated foreground region and the estimated background region from the cropped digital image at the first resolution size.

### Claim 11 (depends on 10)

The system as recited in claim 10, wherein the one or more servers are further configured to cause the system to determine the blended boundary region by determining, utilizing the neural network, the blended boundary region of the object from the cropped digital image at the first resolution size.

### Claim 12 (depends on 10)

The system as recited in claim 10, wherein the one or more servers are further configured to cause the system to:
upsample the cropped digital image and the trimap segmentation to a second resolution size; and
generate the updated foreground region and the updated background region by utilizing the matting algorithm to assign an alpha value to each pixel within the blended boundary region based on the upsampled cropped digital image and the upsampled trimap segmentation at the second resolution size.

### Claim 13 (depends on 10)

The system as recited in claim 10, further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine a position of the cropped digital image relative to the digital image; and
combine the image mask with an additional image mask corresponding to an additional portion of the digital image based on the position of the cropped digital image relative to the digital image.

### Claim 14 (depends on 9)

The system as recited in claim 9, wherein the one or more servers are further configured to cause the system to generate the image mask by:
upsampling an initial image mask generated by the matting algorithm to an original resolution of the digital image; and
refining, utilizing a guided image filter, a boundary between the updated foreground region and the updated background region from the initial image mask at the original resolution of the digital image.

### Claim 15 (depends on 9)

The system as recited in claim 9, wherein the one or more servers are further configured to cause the system to:
generate, for a plurality of digital images, a plurality of predicted blended boundary regions corresponding to objects in the plurality of digital images;
determine a blended boundary loss by comparing the plurality of predicted blended boundary regions and a plurality of ground-truth blended boundary regions for the plurality of digital images; and
learn parameters of the neural network based on the blended boundary loss.

### Claim 16

A computer-implemented method comprising:
extracting a foreground region and a background region of a digital image, wherein the foreground region portrays an object in the digital image;
performing a step for automatically generating a trimap segmentation of the digital image; and
generating, utilizing the trimap segmentation of the digital image, an image mask for the digital image.

### Claim 17 (depends on 16)

The computer-implemented method as recited in claim 16, wherein extracting the foreground region and the background region of the digital image comprises:
cropping the digital image to a portion of the digital image including the object;
resizing the cropped digital image to a first resolution size; and
extracting the foreground region and the background region from the cropped digital image at the first resolution size.

### Claim 18 (depends on 17)

The computer-implemented method as recited in claim 17, further comprising:
upsampling the cropped digital image and the trimap segmentation from the first resolution size to a second resolution size; and
generating an updated foreground region and an updated background region by utilizing a matting algorithm to assign alpha values to portions of a blended boundary region of the trimap segmentation based on the upsampled cropped digital image and the upsampled trimap segmentation.

### Claim 19 (depends on 18)

The computer-implemented method as recited in claim 18, further comprising:
utilizing the matting algorithm to generate an initial image mask comprising the updated foreground region and the updated background region at the second resolution size;
upsampling the initial image mask to an original resolution of the digital image; and
generating the image mask by utilizing a guided image filter to refine a boundary between the updated foreground region and the updated background region from the initial image mask.

### Claim 20 (depends on 17)

The computer-implemented method as recited in claim 17, further comprising combining the image mask of the digital image with an additional image mask corresponding to at least a portion of the digital image according to an original position of the cropped digital image within the digital image.
