# US 2016/0261844 A1 · Methods and Apparatus for Enhancing Depth Maps with Polarization Cues

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0261844 A1
- Document ID: US20160261844A1
- Status: Published application
- Kind: A1
- Application number: 14/979,433
- Filed: 2015-12-27
- Published: 2016-09-08
- Earliest claimed: 2015-03-06
- Assignees: Massachusetts Institute of Technology
- Applicants: Massachusetts Institute of Technology
- Inventors: Achuta Kadambi; Vage Taamazyan; Boxin Shi; Ramesh Raskar
- Web page: https://patentgrep.com/patent/US20160261844A1

## CPC

- G01B 11/24: for measuring contours or curvatures
- G06T 7/507: from shading (G06T7/586 takes precedence)
- G06T 7/514: from specularities
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10141: Special mode during image acquisition
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A 3D imaging system uses a depth sensor to produce a coarse depth map, and then uses the coarse depth map as a constraint in order to correct ambiguous surface normals computed from polarization cues. The imaging system outputs an enhanced depth map that has a greater depth resolution than the coarse depth map. The enhanced depth map is also much more accurate than could be obtained from the depth sensor alone. In many cases, the imaging system extracts the polarization cues from three polarized images. Thus, in many implementations, the system takes only three extra images—in addition to data used to generate the coarse depth map—in order to dramatically enhance the coarse depth map.

## Claims

### Claim 1

A method comprising, in combination:
(a) one or more cameras
(i) capturing a first set of images of an object, which first set of images comprises one or more images, and
(ii) capturing a second set of images of the object, which second set includes a first polarized image, a second polarized image and a third polarized image, such that the first, second and third polarized images capture light that has been filtered at a first polarizer angle, a second polarizer angle and a third polarizer angle, respectively, the first, second and third polarizer angles being different from each other; and
(b) one or more computers
(i) calculating, based on the first set of images, a first depth map of a surface of the object,
(ii) calculating polarization surface normals of the surface, by a computation that involves a Fresnel equation and that is based on the second set of images, and
(iii) calculating, based on the first depth map and the polarization surface normals, a second depth map of the surface;
wherein, for at least one point of the surface, the first depth map indicates a first absolute depth of the point and the second depth map indicates a second absolute depth of the point, the first and second absolute depths being different from each other.

### Claim 2 (depends on 1)

The method of claim 1, wherein the second depth map is substantially more accurate than the first depth map.

### Claim 3 (depends on 1)

The method of claim 1, wherein calculating the second depth map includes solving Equation 13.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first and second set of images are not identical sets.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first depth map is calculated by a computation that does not involve a Fresnel equation.

### Claim 6 (depends on 1)

The method of claim 1, wherein calculating the second depth map involves treating the first depth map as a constraint on polarization normal data regarding the surface.

### Claim 7

A method comprising, in combination:
(a) one or more cameras
(i) capturing a first set of images of an object, which first set of images comprises one or more images, and
(ii) capturing a second set of images of the object, which second set includes a first polarized image, a second polarized image and a third polarized image, such that the first, second and third polarized images capture light that has been filtered at a first polarizer angle, a second polarizer angle and a third polarizer angle, respectively, the first, second and third polarizer angles being different from each other; and
(b) one or more computers
(i) calculating, based on the first set of images, a first depth map of a surface of the object,
(ii) calculating, based on the second set of images, a set of polarization surface normals of the surface,
(ii) calculating, based on the polarization surface normals, a second depth map of the surface,
(iv) calculating a minimum spanning tree of a graph, which graph is weighted based at least in part on degree of polarization of the polarization surface normals,
(v) calculating a binary mask by a computation that includes calculating divergence and that specifies a first region of the first depth map and a second region of the first depth map, the first and second regions being different from each other, and
(vi) modifying the second depth map, such that the modifying is based in part on the minimum spanning tree and in part on enforcing consistency between the second depth map and the first region of the first depth map.

### Claim 8 (depends on 7)

The method of claim 7, wherein the modifying involves computations that solve Equation 13.

### Claim 9 (depends on 7)

The method of claim 7, wherein the graph is weighted in accordance with Equation 10.

### Claim 10 (depends on 7)

The method of claim 7, wherein:
(a) the one or more cameras comprise a set of multiple cameras; and
(b) the position, relative to the scene, of each camera in the set of cameras is different than the position, relative to the scene, of any other camera in the set of cameras.

### Claim 11

Apparatus comprising
(a) one or more cameras for
(i) capturing a first set of images of an object, which first set of images comprises one or more images, and
(ii) capturing a second set of images of the object, which second set includes a first polarized image, a second polarized image and a third polarized image, such that the first, second and third polarized images capture light that has been filtered at a first polarizer angle, a second polarizer angle and a third polarizer angle, respectively, the first, second and third polarizer angles being different from each other; and
(b) one or more computers that are programmed
(i) to calculate, based on the first set of images, a first depth map of a surface of the object,
(ii) to calculate polarization surface normals of the surface, by a computation that involves a Fresnel equation and that is based on the second set of images, and
(iii) to calculate a second depth map of the surface, by a computation that treats the first depth map as a constraint on polarization normal data regarding the surface.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein, under standard testing conditions, the second depth map is substantially more accurate than the first depth map.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein the one or more computers are programmed to solve Equation 13.

### Claim 14

Apparatus comprising, in combination:
(a) one or more cameras for
(i) capturing a first set of images of an object, which first set of images comprises one or more images, and
(ii) capturing a second set of images of the object, which second set includes a first polarized image, a second polarized image and a third polarized image, such that the first, second and third polarized images capture light that has been filtered at a first polarizer angle, a second polarizer angle and a third polarizer angle, respectively, the first, second and third polarizer angles being different from each other; and
(b) one or more computers that are programmed
(i) to calculate, based on the first set of images, a first depth map of a surface of the object,
(ii) to calculate, based on the second set of images, polarization surface normals of the surface,
(ii) to calculate, based on the polarization surface normals, a second depth map of the surface,
(iv) to calculate a minimum spanning tree of a graph, which graph is weighted based at least in part on degree of polarization of the polarization surface normals,
(v) to calculate a binary mask by a computation that includes calculating divergence and that specifies a first region of the first depth map and a second region of the first depth map, the first and second regions being different from each other, and
(vi) to calculate a modification of the second depth map, such that the modification is based in part on the minimum spanning tree and in part on enforcing consistency between the second depth map and the first region of the first depth map.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the one or more computers are programmed to calculate the modification by computations that include solving Equation 13.

### Claim 16 (depends on 14)

The apparatus of claim 14, wherein the one or more computers are programmed to calculate the binary mask in accordance with Equation 7.

### Claim 17 (depends on 14)

The apparatus of claim 14, wherein the one or more computers are programmed to weight the graph in accordance with Equation 10.

### Claim 18 (depends on 14)

The apparatus of claim 14, wherein the one or more computers are programmed:
(a) to perform an initial calculation of azimuth angles of the polarization surface normals, such that the solution of the initial computation for each of the azimuth angles, respectively, has a 180-degree ambiguity, and
(b) to perform an additional calculation that involves minimizing total variation subject to a constraint and that eliminates the 180-degree ambiguity.

### Claim 19 (depends on 14)

The apparatus of claim 14, wherein the one or more computers are programmed, for each respective surface normal, out of the set of polarization surface normals:
(a) to compute a degree of polarization of the respective surface normal; and
(b) to calculate a zenith angle of the respective surface normal
(i) in accordance with Equation 3 if the degree of polarization of the respective normal is less than a threshold, and
(ii) in accordance with Equation 4 if the degree of polarization of the respective surface normal is greater than the threshold.

### Claim 20 (depends on 14)

The apparatus of claim 14, wherein the one or more computers are programmed:
(a) to identify a point of the surface, which point is at a boundary between a first area of the surface and a second area of the surface, which first area has a first spatial frequency and which second area has a second spatial frequency, the first spatial frequency being greater than the second spatial frequency; and
(b) to calculate a 3D shape of the surface such that the 3D shape is continuous at the point.
