# US 2020/0402243 A1 · Video background estimation using spatio-temporal models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0402243 A1
- Document ID: US20200402243A1
- Status: Published application
- Kind: A1
- Application number: 17/011,647
- Filed: 2020-09-03
- Published: 2020-12-24
- Earliest claimed: 2020-09-03
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Itay Benou; Yevgeny Priziment; Tzachi Herskovich
- Web page: https://patentgrep.com/patent/US20200402243A1

## CPC

- G06T 7/254: involving subtraction of images
- G06F 18/251: {of input or preprocessed data}
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/803: {of input or preprocessed data}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques related to video background estimation inclusive of generating a final background picture absent foreground objects based on input video are discussed. Such techniques include generating first and second estimated background pictures using temporal and spatial background picture modeling, respectively, and fusing the first and second estimated background pictures based on first and second confidence maps corresponding to the first and second estimated background pictures to generate the final estimated background picture.

## Claims

### Claim 1

A system for performing video background estimation comprising:
a memory to store an input video picture; and
one or more processors coupled to the memory, the one or more processors to:
apply a temporal background picture model based on the input video picture and one or more temporally prior input video pictures to generate a first estimated background picture and a corresponding first confidence map for the input video picture;
apply a spatial background picture model based on the input video picture to generate a second estimated background picture and a corresponding second confidence map for the input video picture; and
combine the first and second estimated background pictures based on the first and second confidence maps to generate a resultant estimated background picture for the input video picture.

### Claim 2 (depends on 1)

The system of claim 1, the one or more processors further to:
apply foreground object detection to the input video picture to generate a foreground object mask, wherein at least one of generating the first estimated background picture and the first confidence map or generating the second estimated background picture and the second confidence map is further based on the foreground object mask.

### Claim 3 (depends on 1)

The system of claim 1, wherein the first confidence map comprises one of a distance between corresponding pixel values of the first estimated background picture and the input video picture or a variance of a statistical model used to generate the first estimated background picture.

### Claim 4 (depends on 1)

The system of claim 1, wherein generation of the second estimated background picture and the second confidence map comprises application of a neural network layer to an input volume based on the input video picture to generate an output volume comprising the second estimated background picture and the second confidence map.

### Claim 5 (depends on 1)

The system of claim 1, wherein the one or more processors to combine the first and second estimated background pictures based on the first and second confidence maps comprises the one or more processors to select, for each pixel of the resultant estimated background picture, a corresponding pixel value from the first estimated background picture in response to a corresponding value in the first confidence map exceeding a corresponding value in the second confidence map or a corresponding pixel value from the second estimated background picture in response to the corresponding value in the second confidence map exceeding the corresponding value in the first confidence map.

### Claim 6 (depends on 1)

The system of claim 1, wherein the one or more processors to combine the first and second estimated background pictures based on the first and second confidence maps comprises the one or more processors to generate, for each pixel of the resultant estimated background picture, a weighted average of corresponding pixel values from the first and second estimated background pictures weighted using corresponding confidence values from the first and second confidence maps.

### Claim 7 (depends on 1)

The system of claim 1, wherein the one or more processors to combine the first and second estimated background pictures based on the first and second confidence maps comprises the one or more processors to apply a neural network layer to an input volume comprising the first and second estimated background pictures and the first and second confidence maps to generate an output volume comprising the resultant estimated background picture.

### Claim 8 (depends on 1)

The system of claim 1, wherein the one or more processors to combine the first and second estimated background pictures based on the first and second confidence maps comprises the one or more processors to generate, for each pixel of the resultant estimated background picture, a magnitude difference between confidence values of the first and second confidence maps and select one of a pixel value of the first or second estimated background pictures for a pixel value the resultant estimated background picture in response to the magnitude difference exceeding a threshold or generating pixel value the resultant estimated background picture as a weighted average of the pixel values of the first and second estimated background pictures in response to the magnitude difference not exceeding the threshold.

### Claim 9 (depends on 1)

The system of claim 1, wherein the one or more processors to combine the first and second estimated background pictures based on the first and second confidence maps comprises the one or more processors to:
determine an overall temporal confidence value based on the first confidence map and an overall spatial confidence value based on the second confidence map;
determine a magnitude difference between the overall temporal confidence value and the overall spatial confidence value; and
select a gating based fusion in response to the magnitude difference exceeding a threshold or a linear combination fusion in response to the magnitude difference not exceeding the threshold, wherein the one or more processors to combine the first and second estimated background pictures applies the selected one of the gating based fusion or the linear combination fusion.

### Claim 10 (depends on 1)

The system of claim 1, wherein one of the one or more processors to apply the temporal background picture model is further based on the second estimated background picture or the one or more processors to apply the spatial background picture model is further based on the first estimated background picture.

### Claim 11 (depends on 1)

The system of claim 1, wherein the one or more processors to apply the spatial background picture model is further based on the first estimated background picture and the first confidence map, and comprises the one or more processors to determine a pixel value of the second estimated background picture using the spatial background picture model in response to a corresponding confidence value of the first confidence map not exceeding a threshold or to copy a corresponding pixel value of the first estimated background picture to the second estimated background picture in response to the corresponding confidence value of the first confidence map exceeding the threshold.

### Claim 12

A method for performing video background estimation comprising:
applying a temporal background picture model based on an input video picture and one or more temporally prior input video pictures to generate a first estimated background picture and a corresponding first confidence map for the input video picture;
applying a spatial background picture model based on the input video picture to generate a second estimated background picture and a corresponding second confidence map for the input video picture; and
combining the first and second estimated background pictures based on the first and second confidence maps to generate a resultant estimated background picture for the input video picture.

### Claim 13 (depends on 12)

The method of claim 12, wherein combining the first and second estimated background pictures based on the first and second confidence maps comprises selecting, for each pixel of the resultant estimated background picture, a corresponding pixel value from the first estimated background picture in response to a corresponding value in the first confidence map exceeding a corresponding value in the second confidence map or a corresponding pixel value from the second estimated background picture in response to the corresponding value in the second confidence map exceeding the corresponding value in the first confidence map.

### Claim 14 (depends on 12)

The method of claim 12, wherein combining the first and second estimated background pictures based on the first and second confidence maps comprises generating, for each pixel of the resultant estimated background picture, a weighted average of corresponding pixel values from the first and second estimated background pictures weighted using corresponding confidence values from the first and second confidence maps.

### Claim 15 (depends on 12)

The method of claim 12, wherein combining the first and second estimated background pictures based on the first and second confidence maps comprises applying a neural network layer to an input volume comprising the first and second estimated background pictures and the first and second confidence maps to generate an output volume comprising the resultant estimated background picture.

### Claim 16 (depends on 12)

The method of claim 12, wherein one of applying the temporal background picture model is further based on the second estimated background picture or applying the spatial background picture model is further based on the first estimated background picture.

### Claim 17

At least one machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform video background estimation by:
applying a temporal background picture model based on an input video picture and one or more temporally prior input video pictures to generate a first estimated background picture and a corresponding first confidence map for the input video picture;
applying a spatial background picture model based on the input video picture to generate a second estimated background picture and a corresponding second confidence map for the input video picture; and
combining the first and second estimated background pictures based on the first and second confidence maps to generate a resultant estimated background picture for the input video picture.

### Claim 18 (depends on 17)

The machine readable medium of claim 17, wherein combining the first and second estimated background pictures based on the first and second confidence maps comprises selecting, for each pixel of the resultant estimated background picture, a corresponding pixel value from the first estimated background picture in response to a corresponding value in the first confidence map exceeding a corresponding value in the second confidence map or a corresponding pixel value from the second estimated background picture in response to the corresponding value in the second confidence map exceeding the corresponding value in the first confidence map.

### Claim 19 (depends on 17)

The machine readable medium of claim 17, wherein combining the first and second estimated background pictures based on the first and second confidence maps comprises generating, for each pixel of the resultant estimated background picture, a weighted average of corresponding pixel values from the first and second estimated background pictures weighted using corresponding confidence values from the first and second confidence maps.

### Claim 20 (depends on 17)

The machine readable medium of claim 17, wherein one of applying the temporal background picture model is further based on the second estimated background picture or applying the spatial background picture model is further based on the first estimated background picture.
