# US 10,311,574 B2 · Object segmentation, including sky segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,574 B2
- Document ID: US10311574B2
- Status: Granted
- Kind: B2
- Application number: 15/853,206
- Filed: 2017-12-22
- Published: 2018-05-17
- Granted: 2019-06-04
- Earliest claimed: 2016-02-11
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Xiaohui Shen; Zhe Lin; Yi-Hsuan Tsai; Kalyan K. Sunkavalli
- Web page: https://patentgrep.com/patent/US10311574B2

## CPC

- G06T 7/11: Region-based segmentation
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06F 18/295: {Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/82: using neural networks
- G06V 10/85: {Markov-related models; Markov random fields}
- G06V 20/38: {Outdoor scenes}
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20092: Interactive image processing based on input by user
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A digital medium environment includes an image processing application that performs object segmentation on an input image. An improved object segmentation method implemented by the image processing application comprises receiving an input image that includes an object region to be segmented by a segmentation process, processing the input image to provide a first segmentation that defines the object region, and processing the first segmentation to provide a second segmentation that provides pixel-wise label assignments for the object region. In some implementations, the image processing application performs improved sky segmentation on an input image containing a depiction of a sky.

## Claims

### Claim 1

In a digital medium environment including an image processing application that performs object segmentation on an input image, an improved object segmentation method implemented by the image processing application, the method comprising:
receiving an input image that includes an object region to be segmented;
processing the input image to provide a first segmentation that defines the object region, said processing comprising:
parsing the input image to provide a probability mask which classifies individual pixels in the input image;
determining, from a database, multiple images which have layouts at least similar to a layout of the input image, wherein the multiple images include respective masks;
processing the respective masks to provide a weighted average mask; and
combining the probability mask and the weighted average mask to provide the first segmentation.

### Claim 2 (depends on 1)

A method as described in claim 1, wherein the object region to be segmented comprises a depiction of a sky.

### Claim 3 (depends on 1)

A method as described in claim 1, wherein parsing the input image to provide the probability mask comprises parsing the input image using a Conditional Random Field to provide the probability mask.

### Claim 4 (depends on 1)

A method as described in claim 1, wherein determining the multiple images from the database comprises using a Convolutional Neural Network to determine multiple images from the database.

### Claim 5 (depends on 1)

A method as described in claim 1, wherein:
parsing the input image to provide the probability mask comprises using a Conditional Random Field to provide the probability mask;
determining the multiple images from the database comprises using a Convolutional Neural Network to determine the multiple images from the database.

### Claim 6 (depends on 1)

A method as described in claim 1, wherein the object region to be segmented comprises a depiction of a sky, and wherein parsing the input image to provide the probability mask comprises parsing the input image using a Conditional Random Field to provide the probability mask.

### Claim 7 (depends on 1)

A method as described in claim 1, wherein the object region to be segmented comprises a depiction of a sky, and wherein determining the multiple images from the database comprises using a Convolutional Neural Network to determine multiple images from the database.

### Claim 8 (depends on 1)

A method as described in claim 1, wherein the object region to be segmented comprises a depiction of a sky, and wherein:
parsing the input image to provide the probability mask comprises using a Conditional Random Field to provide the probability mask;
determining the multiple images from the database comprises using a Convolutional Neural Network to determine the multiple images from the database.

### Claim 9 (depends on 1)

A method as described in claim 1, further comprising processing the first segmentation to provide a second segmentation, wherein said processing the first segmentation to provide the second segmentation comprises computing a color unary potential and computing a texture unary potential that respectively represent a probability of a pixel in the input image being a part of the object region or not a part of the object region.

### Claim 10 (depends on 1)

A method as described in claim 1, further comprising processing the first segmentation to provide a second segmentation, wherein said processing the first segmentation to provide the second segmentation comprises computing a color unary potential and computing a texture unary potential that represent a probability of a pixel in the input image being a part of the object region or not a part of the object region, wherein the color unary potential is computed using Gaussian Mixture Models and the texture unary potential is computed using a Support Vector Machine.

### Claim 11 (depends on 1)

A method as described in claim 1, further comprising processing the first segmentation to provide a second segmentation, wherein said processing the first segmentation to provide the second segmentation comprises computing a color unary potential and computing a texture unary potential that represent a probability of a pixel in the input image being a part of the object region or not a part of the object region, and computing a pairwise term based on a gradient magnitude between adjacent pixels in the input image.

### Claim 12

In a digital medium environment in which a computing device can use an image processing application to perform object segmentation, one or more computer-readable storage media comprising instructions that are stored thereon that, responsive to execution by the computing device, perform operations comprising:
receiving an input image that includes an object region to be segmented;
processing the input image to provide a first segmentation that defines the object region, said processing comprising:
parsing the input image to provide a probability mask which classifies individual pixels in the input image;
determining multiple images from a database which have layouts at least similar to a layout of the input image, wherein the multiple images include respective masks
processing the respective masks to provide a weighted average mask; and
combining the probability mask and weighted average mask to provide the first segmentation.

### Claim 13 (depends on 12)

The one or more computer-readable storage media as described in claim 12, wherein the object region to be segmented comprises a depiction of a sky.

### Claim 14 (depends on 12)

The one or more computer-readable storage media as described in claim 12, wherein said parsing comprises parsing the input image using a Conditional Random Field to provide a probability mask which classifies individual pixels in the input image.

### Claim 15 (depends on 12)

The one or more computer-readable storage media as described in claim 12, wherein determining the multiple images from the database comprises using a Convolutional Neural Network to determine multiple images from the database.

### Claim 16 (depends on 12)

The one or more computer-readable storage media as described in claim 12, further comprising processing the first segmentation to provide a second segmentation that provides pixel-wise label assignments for the object region, wherein processing the first segmentation to provide the second segmentation comprises computing a color unary potential and computing a texture unary potential that represent a probability of a pixel in the input image being a part of the object region or not a part of the object region, and computing a pairwise term computed based on a gradient magnitude between adjacent pixels in the input image.

### Claim 17 (depends on 16)

The one or more computer-readable storage media as described in claim 16, wherein said processing the first segmentation to provide a second segmentation comprises computing the color unary potential comprises using Gaussian Mixture Models and computing the texture unary potential comprises using a Support Vector Machine.

### Claim 18

A system implemented in a digital medium environment including a computing device having an image processing application to enable object segmentation, the system comprising:
a processing system;
one or more computer readable media storing instructions executable via the processing system to implement an image processing application configured to perform operations comprising:
receiving an input image that includes an object region to be segmented;
processing the input image to provide a segmentation that defines the object region, said processing comprising:
parsing the input image to provide a probability mask which classifies individual pixels in the input image;
determining, from a database, multiple images which have layouts at least similar to a layout of the input image, wherein the multiple images include respective masks;
processing the respective masks to provide a weighted average mask; and
combining the probability mask and the weighted average mask to provide the segmentation.

### Claim 19 (depends on 18)

The system as described in claim 18, wherein:
said parsing is performed using a Conditional Random Field; and
said determining multiple images comprising using a Convolutional Neural Network to determine the multiple image.

### Claim 20 (depends on 18)

The system as described in claim 18, wherein the object region to be segmented comprises a depiction of a sky.