# US 10,122,994 B2 · Object reconstruction from dense light fields via depth from gradients

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,122,994 B2
- Document ID: US10122994B2
- Status: Granted
- Kind: B2
- Application number: 15/799,831
- Filed: 2017-10-31
- Published: 2018-05-17
- Granted: 2018-11-06
- Earliest claimed: 2016-11-11
- Assignees: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich)
- Applicants: Disney Enterprises, Inc.
- Inventors: Kaan Yücer; Changil Kim; Alexander Sorkine-Hornung; Olga Sorkine-Hornung
- Web page: https://patentgrep.com/patent/US10122994B2

## CPC

- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G06T 7/557: from light fields, e.g. from plenoptic cameras
- G06T 7/564: from contours
- G06T 7/593: from stereo images
- G06T 15/205: {Image-based rendering}
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 17/205: {Re-meshing}
- H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597)
- H04N 13/15: for colour aspects of image signals
- H04N 13/232: using fly-eye lenses, e.g. arrangements of circular lenses
- G06T 15/04: Texture mapping
- G06T 2200/21: involving computational photography
- G06T 2207/10052: Images from lightfield camera
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure relates to techniques for reconstructing an object in three dimensions that is captured in a set of two-dimensional images. The object is reconstructed in three dimensions by computing depth values for edges of the object in the set of two-dimensional images. The set of two-dimensional images may be samples of a light field surrounding the object. The depth values may be computed by exploiting local gradient information in the set of two-dimensional images. After computing the depth values for the edges, depth values between the edges may be determined by identifying types of the edges (e.g., a texture edge, a silhouette edge, or other type of edge). Then, the depth values from the set of two-dimensional images may be aggregated in a three-dimensional space using a voting scheme, allowing the reconstruction of the object in three dimensions.

## Claims

### Claim 1

A method for object reconstruction, the method comprising:
receiving a light field represented by a plurality of images, wherein each image is at a different viewpoint, and wherein the plurality of images are associated with an object;
determining a depth for a color edge of the object in an image of the plurality of images;
propagating the depth for the color edge to one or more points around the color edge based on a type of the color edge, wherein a direction of propagation depends on whether the type of color edge is texture or silhouette;
generating a depth map for the image using the depth for the color edge and the depth for the one or more points; and
generating a mesh for the object based on the depth map.

### Claim 2 (depends on 1)

The method of claim 1, wherein the depth for the color edge is propagated in two directions when the color edge is determined to be a texture, and wherein the depth for the color edge is propagated in a single direction when the color edge is determined to be a silhouette.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
identifying a plurality of points along a first line perpendicular to the color edge in the image, wherein the image is a first image;
identifying a second image of the plurality of images, wherein the second image is a similar viewpoint as the first image;
identifying the color edge in the second image;
identifying a plurality of points along a second line perpendicular to the color edge in the second image; and
determining the type of the color edge based on a color of each of the one or more points along the first line and each of the one or more points along the second line.

### Claim 4 (depends on 3)

The method of claim 3, wherein the plurality of points along the first line are a first image gradient, and wherein the plurality of points along the second line are a second image gradient.

### Claim 5 (depends on 3)

The method of claim 3, wherein a first point of the plurality of points along the first line is on a particular side of the color edge in the first image, and wherein a second point of the plurality of points along the second line is on the particular side of the color edge in the second image.

### Claim 6 (depends on 5)

The method of claim 5, wherein the color edge is a first type when the first point and the second point are different colors, and wherein the color edge is a second type when the first point and second point are the same color.

### Claim 7 (depends on 1)

The method of claim 1, wherein the depth map is a first depth map, wherein the mesh is generated by aggregating the first depth map with a second depth map, wherein the second depth map is for an additional image of the plurality of images.

### Claim 8

A non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors, the plurality of instructions when executed by the one or more processors cause the one or more processors to:
receive a light field represented by a plurality of images, wherein each image is at a different viewpoint, and wherein the plurality of images are associated with an object;
determine a depth for a color edge of the object in an image of the plurality of images;
propagate the depth for the color edge to one or more points around the color edge based on a type of the color edge, wherein a direction of propagation depends on whether the type of color edge is texture or silhouette;
generate a depth map for the image using the depth for the color edge and the depth for the one or more points; and
generate a mesh for the object based on the depth map.

### Claim 9 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the depth for the color edge is propagated in two directions when the color edge is determined to be a texture, and wherein the depth for the color edge is propagated in a single direction when the color edge is determined to be a silhouette.

### Claim 10 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the plurality of instructions when executed by the one or more processors further cause the one or more processors to identify a plurality of points along a first line perpendicular to the color edge in the image, wherein the image is a first image;
identify a second image of the plurality of images, wherein the second image is a similar viewpoint as the first image;
identify the color edge in the second image;
identify a plurality of points along a second line perpendicular to the color edge in the second image; and
determine the type of the color edge based on a color of each of the one or more points along the first line and each of the one or more points along the second line.

### Claim 11 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the plurality of points along the first line are a first image gradient, and wherein the plurality of points along the second line are a second image gradient.

### Claim 12 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein a first point of the plurality of points along the first line is on a particular side of the color edge in the first image, and wherein a second point of the plurality of points along the second line is on the particular side of the color edge in the second image.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein the color edge is a first type when the first point and the second point are different colors, and wherein the color edge is a second type when the first point and second point are the same color.

### Claim 14 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the depth map is a first depth map, wherein the mesh is generated by aggregating the first depth map with a second depth map, wherein the second depth map is for an additional image of the plurality of images.

### Claim 15

A system comprising:
one or more processors; and
a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to:
receive a light field represented by a plurality of images, wherein each image is at a different viewpoint, and wherein the plurality of images are associated with an object;
determine a depth for a color edge of the object in an image of the plurality of images;
propagate the depth for the color edge to one or more points around the color edge based on a type of the color edge, wherein a direction of propagation depends on whether the type of color edge is texture or silhouette;
generate a depth map for the image using the depth for the color edge and the depth for the one or more points; and
generate a mesh for the object based on the depth map.

### Claim 16 (depends on 15)

The system of claim 15, wherein the depth for the color edge is propagated in two directions when the color edge is determined to be a texture, and wherein the depth for the color edge is propagated in a single direction when the color edge is determined to be a silhouette.

### Claim 17 (depends on 15)

The system of claim 15, wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to:
identify a plurality of points along a first line perpendicular to the color edge in the image, wherein the image is a first image;
identify a second image of the plurality of images, wherein the second image is a similar viewpoint as the first image;
identify the color edge in the second image;
identify a plurality of points along a second line perpendicular to the color edge in the second image; and
determine the type of the color edge based on a color of each of the one or more points along the first line and each of the one or more points along the second line.

### Claim 18 (depends on 17)

The system of claim 17, wherein the plurality of points along the first line are a first image gradient, and wherein the plurality of points along the second line are a second image gradient.

### Claim 19 (depends on 17)

The system of claim 17, wherein a first point of the plurality of points along the first line is on a particular side of the color edge in the first image, and wherein a second point of the plurality of points along the second line is on the particular side of the color edge in the second image.

### Claim 20 (depends on 19)

The system of claim 19, wherein the color edge is a first type when the first point and the second point are different colors, and wherein the color edge is a second type when the first point and second point are the same color.