# US 10,484,663 B2 · Information processing apparatus and information processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,484,663 B2
- Document ID: US10484663B2
- Status: Granted
- Kind: B2
- Application number: 15/841,431
- Filed: 2017-12-14
- Published: 2018-09-06
- Granted: 2019-11-19
- Earliest claimed: 2017-03-03
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Zhaoxin Li; Chongyu Chen; Lei Zhang
- Web page: https://patentgrep.com/patent/US10484663B2

## CPC

- G06V 10/764: using classification, e.g. of video objects
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24: Classification techniques
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- H04N 13/128: Adjusting depth or disparity
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The disclosure relates to an information processing apparatus and information processing method. The information processing apparatus according to an embodiment includes a processing circuitry configured to: derive a disparity map of an object based on at least two images among multi-view images of the object; estimate confidences of disparity values in the disparity map using a classifier trained in advance; and perform an optimization process on the disparity map based on disparity values of reference pixels having confidences higher than a predetermined level.

## Claims

### Claim 1

An information processing apparatus, comprising:
processing circuitry configured to
derive a disparity map of an object based on at least two images among multi-view images of the object;
estimate confidences of disparity values in the disparity map using a classifier trained in advance; and
perform an optimization process on the disparity map based on disparity values of reference pixels having confidences higher than a predetermined level,
wherein the optimization process further comprises:
triangulating the disparity map based on positions of the reference pixels,
determining an energy function respectively for each of triangular regions with the reference pixels as vertexes, and
optimizing disparity values of pixels in a corresponding triangular region based on the determined energy function.

### Claim 2 (depends on 1)

The information processing apparatus according to claim 1, wherein the optimization process further comprises: dividing the disparity map into regions based on the positions of the reference pixels, and performing the optimization process on different regions in the disparity map independently.

### Claim 3 (depends on 1)

The information processing apparatus according to claim 1, wherein the classifier is obtained based on training images using a least square boosting algorithm.

### Claim 4 (depends on 1)

The information processing apparatus according to claim 1, wherein the deriving the disparity map of the object comprises:
calculating, for a target pixel in a first image among the multi-view images, matching costs between the target pixel and pixels in a predetermined range in a second image among the multi-view images; and
determining a corresponding pixel, of the target pixel, in the second image based on the matching costs, thereby determining a disparity value of the target pixel.

### Claim 5 (depends on 4)

The information processing apparatus according to claim 4, wherein the matching costs are calculated by a block-matching based on zero mean normalized cross correlation.

### Claim 6 (depends on 5)

The information processing apparatus according to claim 5, wherein the training of the classifier and the estimation of the confidences are performed based on one or more of:
a minimum cost of the block-matching;
a difference between a minimum cost and a second minimum cost of the block-matching;
a difference between a disparity value determined for the target pixel in the first image and a disparity value determined for the corresponding pixel in the second image; and
a difference between a disparity value of a pixel and a median disparity value of neighboring pixels of the pixel.

### Claim 7 (depends on 1)

The information processing apparatus according to claim 1, wherein the processing circuitry is further configured to determine a pixel, among pixels having confidences higher than a predetermined level, which satisfies the following condition as the reference pixel:
there are at least a predetermined number of pixels, among the neighboring pixels of the pixel, having disparity differences from the pixel less than a predetermined threshold.

### Claim 8 (depends on 1)

The information processing apparatus according to claim 1, wherein the processing circuitry is further configured to
generate a depth map of the object based on the disparity map on which the optimization process has been performed.

### Claim 9 (depends on 8)

The information processing apparatus according to claim 8, wherein the generating of the depth map comprises: generating the depth map, based on camera parameters, from the disparity map on which the optimization process has been performed.

### Claim 10 (depends on 8)

The information processing apparatus according to claim 8, wherein the processing circuitry is further configured to: identify a face expression, an action or a gesture of the object based on the depth map; and generate a corresponding operation instruction based on the identified face expression, action or gesture.

### Claim 11 (depends on 8)

The information processing apparatus according to claim 8, wherein the processing circuitry is further configured to perform a virtual viewpoint rendering or generate visual information for augmented reality based on the depth map.

### Claim 12 (depends on 8)

The information processing apparatus according to claim 8, wherein the processing circuitry is further configured to:
perform a three-dimensional reconstruction process of the object based on the depth map, wherein the three-dimensional reconstruction process comprises at least one of a generation of a truncated signed distance field, a depth fusion and a texture mapping of the object.

### Claim 13 (depends on 1)

The information processing apparatus according to claim 1, further comprising:
one or more image acquiring devices configured to capture the multi-view images.

### Claim 14

An information processing method, comprising:
deriving a disparity map of an object based on at least two images among multi-view images of the object;
estimating confidences of disparity values in the disparity map using a classifier trained in advance; and
performing an optimization process on the disparity map based on disparity values of reference pixels having confidences higher than a predetermined level, wherein the optimization process further comprises:
triangulating the disparity map based on positions of the reference pixels,
determining an energy function respectively for each of triangular regions with the reference pixels as vertexes, and
optimizing disparity values of pixels in a corresponding triangular region based on the determined energy function.

### Claim 15

An information processing apparatus, comprising:
processing circuitry configured to
derive a disparity map of an object based on at least two images among multi-view images of the object;
estimate confidences of disparity values in the disparity map using a classifier trained in advance;
generate a depth map of the object based on the disparity map;
perform a three-dimensional reconstruction of the object from the depth map based on the confidences;
triangulate the disparity map based on positions of reference pixels;
determine an energy function respectively for each of triangular regions with the reference pixels as vertexes; and
optimize disparity values of pixels in a corresponding triangular region based on the determined energy function.

### Claim 16 (depends on 15)

The information processing apparatus according to claim 15, wherein the three-dimensional reconstruction comprises: generating a truncated signed distance field from the depth map based on the confidences.

### Claim 17 (depends on 16)

The information processing apparatus according to claim 16, wherein a parameter δ which determines a width of a relevant near-surface region during the process of generating the truncated signed distance field is set according to the confidences.

### Claim 18 (depends on 17)

The information processing apparatus according to claim 17, wherein the parameter δ is set as a first value if the confidence is above a predetermined level, and the parameter δ is set as a second value greater than the first value if the confidence is below the predetermined level.

### Claim 19

An information processing method, comprising:
deriving a disparity map of an object based on at least two images among multi-view images of the object;
estimating confidences of disparity values in the disparity map using a classifier trained in advance;
generating a depth map of the object based on the disparity map;
performing a three-dimensional reconstruction of the object from the depth map based on the confidences;
triangulating the disparity map based on positions of reference pixels;
determining an energy function respectively for each of triangular regions with the reference pixels as vertexes; and
optimizing disparity values of pixels in a corresponding triangular region based on the determined energy function.