# US 2012/0250982 A1 · Image processing apparatus, image processing method, program, and recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0250982 A1
- Document ID: US20120250982A1
- Status: Published application
- Kind: A1
- Application number: 13/427,199
- Filed: 2012-03-22
- Published: 2012-10-04
- Earliest claimed: 2011-03-31
- Assignees: Sony Corporation
- Inventors: Masato ITO; Kohtaro Sabe; Jun Yokono
- Web page: https://patentgrep.com/patent/US20120250982A1

## CPC

- G06T 7/73: using feature-based methods
- G06T 7/12: Edge-based segmentation
- G06T 7/174: involving the use of two or more images
- G06T 7/194: involving foreground-background segmentation
- G06T 2207/20081: Training; Learning
- G06T 2207/20164: Salient point detection; Corner detection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An image processing apparatus includes: an image feature outputting unit that outputs each of image features in correspondence with a time of the frame; a foreground estimating unit that estimates a foreground image at a time s by executing a view transform as a geometric transform on a foreground view model and outputs an estimated foreground view; a background estimating unit that estimates a background image at the time s by executing a view transform as a geometric transform on a background view model and outputs an estimated background view; a synthesized view generating unit that generates a synthesized view by synthesizing the estimated foreground and background views; a foreground learning unit that learns the foreground view model based on an evaluation value; and a background learning unit that learns the background view model based on the evaluation value by updating the parameter of the foreground view model.

## Claims

### Claim 1

An image processing apparatus comprising:
an image feature outputting unit that outputs each of image features, which are formed as features of a plurality of feature points of images of each frame in data of an input moving image, in correspondence with a time of the frame;
a foreground estimating unit that estimates a foreground image at a time s by executing a view transform as a geometric transform on a foreground view model having the image feature of a foreground image in the image as a parameter in regard to the image feature at the time s, and then outputs an estimated foreground view;
a background estimating unit that estimates a background image at the time s by executing a view transform as a geometric transform on a background view model having the image feature of a background image in the image as a parameter in regard to the image feature at the time s, and then outputs an estimated background view;
a synthesized view generating unit that generates a synthesized view by synthesizing the estimated foreground view and the estimated background view;
a foreground learning unit that learns the foreground view model based on an evaluation value obtained by comparison between the synthesized view and the image feature at the time s by updating the parameter of the foreground view model based on a stochastic generation model; and
a background learning unit that learns the background view model based on the evaluation value by updating the parameter of the background view model based on a stochastic generation model.

### Claim 2 (depends on 1)

The image processing apparatus according to claim 1, wherein the foreground estimating unit and the background estimating unit each
calculate a posterior probability of each of a plurality of the view transforms based on the evaluation value,
estimate the view transform based on the image feature and the posterior probability, and
output the estimated foreground view and the estimated background view, respectively, by executing the view transforms on the view model.

### Claim 3 (depends on 1)

The image processing apparatus according to claim 1, wherein the foreground estimating unit and the background estimating unit each
weight the parameter of the view model subjected to a plurality of the view transforms based on a posterior probability of each of the view transforms calculated based on the evaluation value, and
update the parameter of the foreground view model and the parameter of the background view model, respectively, based on the weighted parameter.

### Claim 4 (depends on 1)

The image processing apparatus according to claim 1, wherein the view model is configured as a multi-view model formed by a plurality of view models corresponding to images obtained by viewing one of the foreground and the background at different angles.

### Claim 5 (depends on 1)

The image processing apparatus according to claim 1, wherein one of the foreground view model and the background view model is configured by a multi-view model formed by a plurality of view models corresponding to images obtained by viewing one of the foreground and the background at different angles.

### Claim 6 (depends on 4)

The image processing apparatus according to claim 4, wherein the foreground estimating unit and the background estimating unit each
calculate a posterior probability of each of a plurality of the view models based on the evaluation value,
estimate the view transform based on the image feature and the posterior probability, and
output the estimated foreground view and the estimated background view by executing the view transform on the estimated view model corresponding to transition,
wherein one of HMM, FNN, RNN, a particle filter, and a Kalman filter is used dynamics learning estimation model used for estimating a prior probability based on view transition in calculation of a posterior probability of the view model.

### Claim 7 (depends on 1)

The image processing apparatus according to claim 1, wherein the foreground estimating unit and the background estimating unit each
calculate a posterior probability of each of a plurality of the view transforms based on the evaluation value, and
estimate the view transform based on the image feature and the posterior probability.

### Claim 8 (depends on 1)

The image processing apparatus according to claim 1, wherein the foreground learning unit and the background learning unit each
weight the parameter of the view model corresponding to a plurality of transitions based on a posterior probability of each of the transitions calculated based on the evaluation value,
weight the parameter of the view model subjected to a plurality of the view transforms based on a posterior probability of each of the view transforms calculated based on the evaluation value, and
update the parameter of the foreground view model and the parameter of the background view model, respectively, based on the weighted parameter.

### Claim 9 (depends on 1)

The image processing apparatus according to claim 1, wherein one of HMM, FNN, RNN, a particle filter, and a Kalman filter is used as a dynamics learning estimation model used for estimating a prior probability based on view transform motion in calculation of a posterior probability of the view transform in the stochastic generation model.

### Claim 10 (depends on 1)

The image processing apparatus according to claim 1, wherein the stochastic generation model used in the foreground learning unit is different from the stochastic generation model used in the background learning unit.

### Claim 11 (depends on 1)

The image processing apparatus according to claim 1, wherein the image feature outputting unit outputs information in which a pixel position and a pixel value of each pixel correspond to each other as the image feature.

### Claim 12 (depends on 1)

The image processing apparatus according to claim 1, wherein the image feature outputting unit outputs a set of feature point positions detected by a Harris corner detection method as the image feature.

### Claim 13

An image processing method comprising:
outputting, by an image feature outputting unit, each of image features, which are formed as features of a plurality of feature points of images of each frame in data of an input moving image, in correspondence with a time of the frame;
estimating, by a foreground estimating unit, a foreground image at a time s by executing a view transform as a geometric transform on a foreground view model having the image feature of a foreground image in the image as a parameter in regard to the image feature at the time s, and then outputting an estimated foreground view;
estimating, by a background estimating unit, a background image at the time s by executing a view transform as a geometric transform on a background view model having the image feature of a background image in the image as a parameter in regard to the image feature at the time s, and then outputting an estimated background view;
generating, by a synthesized view generating unit, a synthesized view by synthesizing the estimated foreground view and the estimated background view;
learning, by a foreground learning unit, the foreground view model based on an evaluation value obtained by comparison between the synthesized view and the image feature at the time s by updating the parameter of the foreground view model based on a stochastic generation model; and
learning, by a background learning unit, the background view model based on the evaluation value by updating the parameter of the background view model based on a stochastic generation model.

### Claim 14

A program causing a computer to function as an image processing apparatus including:
an image feature outputting unit that outputs each of image features, which are formed as features of a plurality of feature points of images of each frame in data of an input moving image, in correspondence with a time of the frame;
a foreground estimating unit that estimates a foreground image at a time s by executing a view transform as a geometric transform on a foreground view model having the image feature of a foreground image in the image as a parameter in regard to the image feature at the time s, and then outputs an estimated foreground view;
a background estimating unit that estimates a background image at the time s by executing a view transform as a geometric transform on a background view model having the image feature of a background image in the image as a parameter in regard to the image feature at the time s, and then outputs an estimated background view;
a synthesized view generating unit that generates a synthesized view by synthesizing the estimated foreground view and the estimated background view;
a foreground learning unit that learns the foreground view model based on an evaluation value obtained by comparison between the synthesized view and the image feature at the time s by updating the parameter of the foreground view model based on a stochastic generation model; and
a background learning unit that learns the background view model based on the evaluation value by updating the parameter of the background view model based on a stochastic generation model.

### Claim 15 (depends on 14)

A recording medium storing the program according to claim 14.
