US 10,593,043 B2 · Utilizing deep learning for boundary-aware image segmentation

Abstract

Systems and methods are disclosed for segmenting a digital image to identify an object portrayed in the digital image from background pixels in the digital image. In particular, in one or more embodiments, the disclosed systems and methods use a first neural network and a second neural network to generate image information used to generate a segmentation mask that corresponds to the object portrayed in the digital image. Specifically, in one or more embodiments, the disclosed systems and methods optimize a fit between a mask boundary of the segmentation mask to edges of the object portrayed in the digital image to accurately segment the object within the digital image.

Claims20

Dependent claims

In a digital medium environment for editing digital visual media, a computer-implemented method of using deep learning to segment objects from the digital visual media, the computer-implemented method comprising:

generating a probability map for an input image, wherein the probability map indicates object pixels predicted to correspond to an object;
generating a boundary map for the input image, wherein the boundary map indicates edge pixels predicted to correspond to edges of the object;
combining the probability map and the boundary map;
and
generating a segmentation mask that indicates a boundary of the object by iteratively adjusting a mask boundary of the object based on the combination of the boundary map and the probability map.

2 more elements

A system comprising:

a memory comprising:
a convolutional neural network trained to generate feature maps of digital images;
a first de-convolutional neural network trained to generate probability maps indicating pixels of objects portrayed within digital images;
a second de-convolutional neural network trained to generate boundary maps indicating pixels of boundaries of objects portrayed within digital images;
at least one processor;
and
a non-transitory computer readable medium storing instructions thereon that, when executed by the at least one processor, cause the system to:
generate, using the convolutional neural network, a feature map for an input image, the input image comprising an object portrayed therein;
generate, based on the feature map and utilizing the first de-convolutional neural network, a probability map for the input image;
generate, based on the feature map and utilizing the second de-convolutional neural network to a boundary map for the input image;
and
generate, based on the probability map and the boundary map, a segmentation mask comprising an identification of boundary pixels within the input image corresponding to the object.

9 more elements

A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computer system to:

generate, using a convolutional neural network, a feature map for an input image, the input image comprising an object portrayed within the input image;
utilize a first de-convolutional neural network to generate, based on the feature map, a probability map for the input image, wherein the probability map indicates object pixels predicted to correspond to the object;
utilize a second de-convolutional neural network to generate, based on the feature map, a boundary map for the input image, wherein the boundary map indicates edge pixels predicted to correspond to edges of the object;
and
generate, based on the probability map and the boundary map, a segmentation mask comprising an identification of boundary pixels within the input image corresponding to the object.

2 more elements

Dependencies are inferred from the claim text.

Utilizing deep learning for boundary-aware image segmentation

Life of this patent

Utilizing deep learning for boundary-aware image segmentation

Life of this patent

Abstract

Claims20