# US 9,437,008 B1 · Image segmentation using bayes risk estimation of scene foreground and background

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,437,008 B1
- Document ID: US9437008B1
- Status: Granted
- Kind: B1
- Application number: 14/804,444
- Filed: 2015-07-21
- Granted: 2016-09-06
- Earliest claimed: 2015-07-21
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Hiroyuki Takeda; Mohammad Gharavi-Alkhansari
- Web page: https://patentgrep.com/patent/US9437008B1

## CPC

- G06T 7/11: Region-based segmentation
- G06F 18/24155: {Bayesian classification}
- G06T 7/143: involving probabilistic approaches, e.g. Markov random field [MRF] modelling
- G06T 7/194: involving foreground-background segmentation
- G06V 10/28: Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
- G06V 10/56: relating to colour
- G06V 10/764: using classification, e.g. of video objects
- G06V 40/162: {using pixel segmentation or colour matching}
- G06V 40/167: {using comparisons between temporally consecutive images}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Enhancing segmentation of objects within an image by utilizing an error cost value. Each error cost value comprises a cost value Cfb if segmentation decides on foreground for an element that should be background, and a cost value Cbf if segmentation decides on background for an element that should be in the foreground. Different areas of an image are estimated and error cost values assigned for a definite foreground (very high Cbf and very low Cfb), a definite background (very high Cfb and very low Cbf), an initial foreground (high Cbf and low Cfb), and an initial background (high Cfb and low Cbf). These error costs are utilized in combination with conventional segmentation decisions to modify those segmentation decisions and thus enhance segmentation accuracy.

## Claims

### Claim 1

An apparatus for segmenting an object from within an image, comprising:
(a) at least one processor configured for processing of a received image; and
(b) memory storing instructions;
(c) said instructions when executed by the processor performing steps comprising:
(i) receiving an image, comprising pixels, and at least a selection point for an object being selected in said image to which image segmentation is to be performed, within an electronic device configured for performing image processing;
(ii) quantizing the image to obtain information utilized in selecting area in pre-determined areas, comprising areas of definite foreground, definite background, initial foreground, initial background;
(iii) estimating areas of the image covered by the definite foreground, the definite background, the initial foreground, and the initial background;
(iv) assigning sets of risk values to each of said definite foreground, the definite background, the initial foreground, and the initial background;
(v) wherein each of said risk values comprises a cost value Cfb if segmentation decides on foreground for an element that should be background, and a cost value Cbf if segmentation decides on background for an element that should be in the foreground;
(vi) wherein for said initial foreground, a cost value Cbf is assigned which is higher than cost value Cfb, while for said initial background a cost value Cfb is assigned which is higher than cost value Cbf;
(vii) wherein for said definite foreground, a cost value Cbf is assigned which is significantly larger than cost value Cfb, while for said definite background a cost value Cfb is assigned which is significantly higher than cost value Cbf with a difference between cost values assigned to the definite foreground and definite background being significantly larger than assigned to the initial foreground and initial background; and
(viii) making segmentation decisions for the pixels in the image based on information from that image, and modifying those segmentation decisions based on the cost values of the area in which the pixels are found.

### Claim 2 (depends on 1)

The apparatus as recited in claim 1, wherein said apparatus comprises an image capture device selected from the group of still and video image capture devices consisting of cameras, cellular phones, pads, tablets, laptops, notebooks and surveillance equipment.

### Claim 3 (depends on 1)

The apparatus as recited in claim 1, wherein said instructions when executed by the processor further performing steps comprising utilizing a known object detection process for estimating the extent of said definite foreground, and said initial foreground.

### Claim 4 (depends on 2)

The apparatus as recited in claim 2, wherein said known object detection process comprises a human object detection process.

### Claim 5 (depends on 1)

The apparatus as recited in claim 1, wherein said instructions when executed by the processor are configured for making said segmentation decisions as determined in response to information about the image as selected from the group of available image information consisting of depth information, contrast mapping, and texture mapping.

### Claim 6 (depends on 1)

The apparatus as recited in claim 1, wherein assigning of risk values can be determined by a user.

### Claim 7 (depends on 6)

The apparatus as recited in claim 6, wherein said instructions when executed by the processor further perform steps comprising making an initial presumption about the object area based on contact area of said selection point detected on a touch screen input.

### Claim 8 (depends on 1)

The apparatus as recited in claim 1, wherein said selection point for an object being selected comprises a touch screen input.

### Claim 9 (depends on 1)

The apparatus as recited in claim 1, wherein said instructions when executed by the processor are configured to assess overall risk R as:
R=Cfbp(Hfore|Hback)p(Hback)+Cbfp(Hback|Hfore)p(Hfore)
where H is a hypothesis, Hfore and Hback is a label of the pixel either foreground or background, with function p( ) indicating probability which can be any real value between 0 and 1, so that value p(Hfore) and p(Hback) are prior probability, and p(Hfore|Hback) and p(Hback|Hfore) are conditional probability.

### Claim 10 (depends on 9)

The apparatus as recited in claim 9, wherein said instructions when executed by the processor are configured for minimizing overall risk R by modifying a classifier and deciding unknown pixel x being in the foreground if
C
b
⁢
⁢
f
⁢
p
⁡
(
H
back
|
H
fore
)
⁢
p
⁡
(
H
fore
)
C
f
⁢
⁢
b
⁢
p
⁡
(
H
fore
|
H
back
)
⁢
p
⁡
(
H
back
)
>
1.

### Claim 11

An apparatus for segmenting an object from within an image, comprising:
(a) at least one processor configured for processing of a received image; and
(b) memory storing instructions;
(c) said instructions when executed by the processor performing steps comprising:
(i) receiving an image, comprising pixels, and at least a selection point for an object being selected in said image to which image segmentation is to be performed, within an electronic device configured for performing image processing;
(ii) receiving additional image information comprising depth information, or contrast mapping, or texture mapping, or any combination of this information, upon which to assess risk and make segmentation decisions;
(iii) quantizing the image to obtain information utilized in selecting area in pre-determined areas, comprising areas of definite foreground, definite background, initial foreground, initial background;
(iv) estimating areas of the image covered by the definite foreground, by the definite foreground, the definite background, the initial foreground, and the initial background;
(v) assigning sets of risk values to each of said definite foreground, the definite background, the initial foreground, and the initial background;
(vi) wherein each of said risk values comprises a cost value Cfb if segmentation decides on foreground for an element that should be background, and a cost value Cbf if segmentation decides on background for an element that should be in the foreground;
(vii) wherein for said initial foreground, a cost value Cbf is assigned which is higher than cost value Cfb, while for said initial background a cost value Cfb is assigned which is higher than cost value Cbf;
(viii) wherein for said definite foreground, a cost value Cbf is assigned which is significantly larger than cost value Cfb, while for said definite background a cost value Cfb is assigned which is significantly higher than cost value Cbf with a difference between cost values assigned to the definite foreground and definite background being significantly larger than assigned to the initial foreground and initial background; and
(ix) making segmentation decisions for the pixels in the image based on information from that image, and modifying those segmentation decisions based on the cost values of the area in which the pixels are found.

### Claim 12

A method of segmenting an object from within an image, comprising:
(a) receiving an image, comprising pixels, and at least a selection point for an object being selected in said image to which image segmentation is to be performed, within an electronic device configured for performing image processing;
(b) quantizing the image to obtain information utilized in selecting area in pre-determined areas, comprising areas of definite foreground, definite background, initial foreground, initial background;
(c) estimating areas of the image covered by the definite foreground, the definite background, the initial foreground, and the initial background;
(d) assigning sets of risk values to each of said definite foreground, the definite background, the initial foreground, and the initial background;
(e) wherein each of said risk values comprises a cost value Cfb if segmentation decides on foreground for an element that should be background, and a cost value Cbf if segmentation decides on background for an element that should be in the foreground;
(f) wherein for said initial foreground, a cost value Cbf is assigned which is higher than cost value Cfb, while for said initial background a cost value Cfb is assigned which is higher than cost value Cbf;
(g) wherein for said definite foreground, a cost value Cbf is assigned which is significantly larger than cost value Cfb, while for said definite background a cost value Cfb is assigned which is significantly higher than cost value Cbf with a difference between cost values assigned to the definite foreground and definite background being significantly larger than assigned to the initial foreground and initial background; and
(h) making segmentation decisions for the pixels in the image based on information from that image, and modifying those segmentation decisions based on the cost values of the area in which the pixels are found.

### Claim 13 (depends on 12)

The method as recited in claim 12, further comprising utilizing a known object detection process for estimating the extent of said definite foreground, and said initial foreground.

### Claim 14 (depends on 13)

The method as recited in claim 13, wherein said known object detection process comprises a human object detection process.

### Claim 15 (depends on 12)

The method as recited in claim 12, wherein said segmentation decisions are further determined in response to information about the image selected from the group of available image information consisting of depth information, contrast mapping, texture mapping.

### Claim 16 (depends on 12)

The method as recited in claim 12, wherein assigning of risk values is determined by a user.

### Claim 17 (depends on 12)

The method as recited in claim 12, wherein said selection point for an object being selected comprises a touch screen input, and making an initial presumption about the object area based on contact area of said selection point detected on the touch screen input.

### Claim 18 (depends on 12)

The method as recited in claim 12, wherein overall risk R is defined as: R=Cfbp(Hfore|Hback)p(Hback)+Cbfp(Hback|Hfore)p(Hfore)
where H is a hypothesis, Hfore and Hback is a label of the pixel either foreground or background, with function p( ) indicating probability which can be any real value between 0 and 1, so that value p(Hfore) and p(Hback) are prior probability, and p(Hfore|Hback) and p(Hback|Hfore) are conditional probability.

### Claim 19 (depends on 18)

The method as recited in claim 18, wherein overall risk R is minimized by modifying a classifier and deciding unknown pixel x being in the foreground if
C
b
⁢
⁢
f
⁢
p
⁡
(
H
back
|
H
fore
)
⁢
p
⁡
(
H
fore
)
C
f
⁢
⁢
b
⁢
p
⁡
(
H
fore
|
H
back
)
⁢
p
⁡
(
H
back
)
>
1.

### Claim 20 (depends on 12)

The method as recited in claim 12, wherein said electronic device is selected from the group of imaging devices consisting of cameras, cellular phones, pads, tablets, laptops, notebooks and surveillance equipment.