# US 2019/0172212 A1 · Multi-modal data fusion for scene segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0172212 A1
- Document ID: US20190172212A1
- Status: Published application
- Kind: A1
- Application number: 15/995,536
- Filed: 2018-06-01
- Published: 2019-06-06
- Earliest claimed: 2017-12-06
- Applicants: Blueprint Reality Inc.
- Inventors: Benjamin James Sheftel; Tryon Williams
- Web page: https://patentgrep.com/patent/US20190172212A1

## CPC

- G06T 7/194: involving foreground-background segmentation
- G06T 7/136: involving thresholding
- G06T 7/143: involving probabilistic approaches, e.g. Markov random field [MRF] modelling
- G06T 7/155: involving morphological operators
- H04N 9/75: Chroma key
- G06T 2207/20044: Skeletonization; Medial axis transform
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Multiple detection techniques are used to determine whether the individual pixels in an image are background or foreground. The outputs from the multiple techniques are combined according to a confidence metric associated with each technique, which may vary from pixel to pixel. The resulting blended value for each pixel is compared with a threshold to determine whether the pixel is foreground or background.

## Claims

### Claim 1

A method for segmenting an image into foreground and background comprising the steps of:
capturing an image of a scene;
for each of multiple detection techniques, processing the image to preliminarily determine that each pixel in the image is either background or foreground;
for each pixel, combining the preliminary determinations in proportion to a confidence level attributed to each detection technique to yield a blended value for each pixel, wherein the blended value lies in a range varying from a value that represents background to a value that represents foreground;
defining a threshold value that lies in the range; and
for each pixel, determining that the pixel is foreground or background depending on which side of the threshold the blended value lies.

### Claim 2 (depends on 1)

The method of claim 1, wherein the value that represents background is 0 and the value that represents foreground is 1.

### Claim 3 (depends on 1)

The method of claim 1, wherein the multiple detection techniques include two or more of:
skeletal pose determination;
chromakeying;
static subtraction;
global depth comparison;
depth map comparison;
reference to a 3D model of a background in the scene;
a tracked device envelope;
object recognition with computer vision; and
continuity detection.

### Claim 4 (depends on 1)

The method of claim 1, wherein the preliminary determinations are combined in proportion to a weighting factor corresponding to each detection technique.

### Claim 5 (depends on 4)

The method of claim 4, wherein the weighting factors and confidence levels are specific to each pixel.

### Claim 6 (depends on 1)

The method of claim 1, further comprising defining an order in which the preliminary determinations of the detection techniques are to be combined, wherein the preliminary determinations are combined in the order in a cascading confidence determination.

### Claim 7 (depends on 6)

The method of claim 6, wherein the cascading confidence determination comprises, for each pixel:
initializing a working value y of the blended value to be equal to 1;
initializing a working value ctotal of an aggregate confidence value to be equal to 0;
for each detection technique numbered i, in said order:
modifying y to equal y+(yi−y)*(1−ctotal), wherein yi is the preliminary determination from the detection technique numbered i; and
modifying ctotal to equal ctotal+ci*(1−ctotal), wherein ci is the confidence level attributed to the detection technique numbered i; and
setting the blended value equal to y after all said modifying steps.

### Claim 8 (depends on 1)

The method of claim 1, performed in real-time.

### Claim 9 (depends on 1)

The method of claim 1, further comprising removing the background pixels from the image.

### Claim 10 (depends on 9)

The method of claim 9, further comprising compositing the image with the background pixels removed onto a further image.

### Claim 11 (depends on 1)

The method of claim 1, wherein the image is one of multiple frames in a video.

### Claim 12 (depends on 11)

The method of claim 11, wherein a determination that a pixel is foreground is changed to a determination that the pixel is background if the pixel has not been determined as foreground in an immediately preceding frame.

### Claim 13 (depends on 1)

The method of claim 1, wherein the image is captured by a camera on a drone.

### Claim 14

A system for segmenting an image into foreground and background comprising:
a camera configured to capture an image of a scene; and
a processor connected to the camera and configured to:
receive the image from the camera;
for each of multiple detection techniques, process the image to preliminarily determine that each pixel in the image is either background or foreground;
for each pixel, combine the preliminary determinations in proportion to a confidence level attributed to each detection technique to yield a blended value for each pixel, wherein the blended value lies in a range varying from a value that represents background to a value that represents foreground;
define a threshold value that lies in the range; and
for each pixel, determine that the pixel is foreground or background depending on which side of the threshold the blended value lies.

### Claim 15 (depends on 14)

The system of claim 14, wherein the processor determines that the pixels are foreground or background in real-time.

### Claim 16 (depends on 14)

The system of claim 14, wherein the processor is further configured to remove the background pixels from the image.

### Claim 17 (depends on 16)

The system of claim 16, wherein the processor is further configured to composite the image with the background pixels removed onto a further image.

### Claim 18 (depends on 14)

The system of claim 14, wherein the camera is on a drone.
