# US 2019/0379873 A1 · Multimodal foreground background segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0379873 A1
- Document ID: US20190379873A1
- Status: Published application
- Kind: A1
- Application number: 16/214,027
- Filed: 2018-12-07
- Published: 2019-12-12
- Earliest claimed: 2013-04-15
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Alvaro Collet Romea; Bao Zhang; Adam G. Kirk
- Web page: https://patentgrep.com/patent/US20190379873A1

## CPC

- H04N 9/75: Chroma key
- G01B 11/22: for measuring depth
- G06F 18/256: {of results relating to different input data, e.g. multimodal recognition}
- G06T 7/11: Region-based segmentation
- G06T 7/136: involving thresholding
- G06T 7/174: involving the use of two or more images
- G06T 7/194: involving foreground-background segmentation
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/593: from stereo images
- G06T 7/596: {from three or more stereo images}
- G06V 10/28: Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
- G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition}
- G06V 10/98: Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597)
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10048: Infrared image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The subject disclosure is directed towards a framework that is configured to allow different background-foreground segmentation modalities to contribute towards segmentation. In one aspect, pixels are processed based upon RGB background separation, chroma keying, IR background separation, current depth versus background depth and current depth versus threshold background depth modalities. Each modality may contribute as a factor that the framework combines to determine a probability as to whether a pixel is foreground or background. The probabilities are fed into a global segmentation framework to obtain a segmented image.

## Claims

### Claim 1

A system comprising, a foreground background segmentation framework, including a multimodal segmentation algorithm configured to accept contribution factors from different segmentation modalities and process the contribution factors to determine foreground versus background data for each element of an image that is useable to determine whether that element is a foreground or background element.

### Claim 2 (depends on 1)

The system of claim 1 wherein at least one element comprises a pixel.

### Claim 3 (depends on 1)

The system of claim 1 wherein the foreground versus background data comprises a probability score.

### Claim 4 (depends on 1)

The system of claim 1 wherein the different segmentation modalities correspond to any of: a red, green blue (RGB) background subtraction, chroma keying, infrared (IR) background subtraction, a current computed depth versus previously computed background depth evaluation, or a current depth versus threshold depth evaluation.

### Claim 5 (depends on 1)

The system of claim 1 wherein the foreground background segmentation framework is further configured to output the foreground versus background data for each element to a global binary segmentation algorithm.

### Claim 6 (depends on 1)

The system of claim 1 wherein the framework is configured to apply a weight for each contribution factor.

### Claim 7 (depends on 6)

The system of claim 6 wherein the framework is configured to select a weight set from among a plurality of weight sets to apply the weight for each contribution factor.

### Claim 8 (depends on 6)

The system of claim 6 wherein the framework is coupled to a multiple camera set environment, and wherein the framework is configured to apply a weight set to one camera set that is different from a weight set applied to another camera set.

### Claim 9 (depends on 1)

The system of claim 1 wherein the framework is coupled to a multiple camera set environment, and wherein the framework is configured to determine the foreground versus background data based on zero or more contribution factors in conjunction with information that corresponds to other camera foreground versus background data.

### Claim 10 (depends on 1)

The system of claim 1 wherein the framework is configured to determine the foreground versus background data based on zero or more contribution factors and detection information processed from an image.

### Claim 11

A method, comprising, processing a frame of image data and processing depth data computed from a corresponding depth-related image, including performing background subtraction on an element of the image data to obtain a background subtraction contribution factor for that element, determining one or more other depth-based contribution factors based upon the depth data associated with that element, computing a combined data term based at least in part upon a contribution from the background contribution factor and a contribution from each of the one or more other depth-based contribution factors, and using the data term in conjunction with other data terms as input to a global binary segmentation mechanism to obtain a segmented image.

### Claim 12 (depends on 11)

The method of claim 11 further comprising processing a frame of image data using chroma keying to obtain a chroma keying contribution factor, for the element and wherein computing the combined data term further comprises using a contribution from the chroma keying contribution factor.

### Claim 13 (depends on 11)

The method of claim 11 wherein performing the background subtraction on an element of the image data comprises performing infrared background subtraction using captured infrared image data for a current element and previously captured background infrared image data.

### Claim 14 (depends on 11)

The method of claim 11 wherein determining the one or more other depth-based contribution factors comprises evaluating a difference between currently captured depth data corresponding to the element and previously captured background depth data corresponding to the element.

### Claim 15 (depends on 11)

The method of claim 11 wherein determining the one or more other depth-based contribution factors comprises evaluating currently captured depth data corresponding to the element and threshold depth data

### Claim 16 (depends on 11)

The method of claim 11 further comprising, using information corresponding to background versus foreground information corresponding to at least one other cameras as in computing the combined data term.

### Claim 17

One or more machine-readable storage media or logic having executable instructions, which when executed perform steps, comprising:
(a) selecting a pixel as a selected pixel;
(b) processing pixel data, including:
processing red, green and blue (RGB) pixel data of one or more images to determine one or more RGB contributing factors indicative of whether the selected pixel is likely a foreground or background pixel in a current image;
processing infrared (IR) pixel data of one or more infrared images to determine one or more IR contributing factors indicative of whether the selected pixel is likely a foreground or background pixel in the current image;
processing pixel depth data to determine one or more depth-based contributing factors indicative of whether the selected pixel is likely a foreground or background pixel in the current image;
(c) combining the contributing factors into a data term for the selected pixel;
(d) maintaining the data term for the selected pixel independent of other data terms for any other pixels;
(e) selecting a different pixel as the selected pixel; and
(f) returning to step (b) for a plurality of pixels to obtain a plurality of data terms.

### Claim 18 (depends on 17)

The one or more machine-readable storage media or logic of claim 17 wherein processing the RGB pixel data of the one or more images to determine the one or more RGB contributing factors comprises performing at least one of: background subtraction based on a previous RGB background image and a current RGB image, or performing chroma keying based on known background data and a current RGB image.

### Claim 19 (depends on 17)

The one or more machine-readable storage media or logic of claim 17 wherein processing the IR pixel data of the one or more images to determine the one or more IR contributing factors comprises performing background subtraction based on a previous IR background image and a current IR image.

### Claim 20 (depends on 17)

The one or more machine-readable storage media or logic of claim 17 wherein processing the pixel depth data to determine the one or more depth-based contributing factors comprises performing at least one of: evaluating current pixel depth data against previous background pixel data, or evaluating current pixel depth data against threshold depth data.
