# US 2021/0241475 A1 · Method and system for optimizing depth imaging

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241475 A1
- Document ID: US20210241475A1
- Status: Published application
- Kind: A1
- Application number: 17/050,881
- Filed: 2019-05-03
- Published: 2021-08-05
- Earliest claimed: 2018-05-03
- Assignees: The Governing Council of the University of Toronto
- Applicants: The Governing Council of the University of Toronto
- Inventors: Kiriakos Neoklis KUTULAKOS; Seyed Parsa MIRDEHGHAN; Wenzheng CHEN
- Web page: https://patentgrep.com/patent/US20210241475A1

## CPC

- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G01B 11/22: for measuring depth
- G01B 11/2513: {with several lines being projected in more than one direction, e.g. grids, patterns}
- H04N 13/207: using a single two-dimensional [2D] image sensor
- H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

There is provided a system and method for optimizing depth imaging. The method including: illuminating one or more scenes with illumination patterns; capturing one or more images of each of the scenes; reconstructing the scenes; estimating the reconstruction error and a gradient of the reconstruction error; iteratively performing until the reconstruction error reaches a predetermined error condition: determining a current set of control vectors and current set of reconstruction parameters; illuminating the one or more scenes with the illumination patterns governed by the current set of control vectors; capturing one or more images of each of the scenes while the scene is being illuminated with at least one of the illumination patterns; reconstructing the scenes from the one or more captured images using the current reconstruction parameters; and estimating an updated reconstruction error and gradient; and outputting at least one of control vectors and reconstruction parameters.

## Claims

### Claim 1

A computer-implemented method for optimizing depth imaging, the method comprising:
illuminating one or more scenes with illumination patterns governed by an initial set of control vectors;
capturing one or more images of each of the scenes while the scene is being illuminated with at least one of the illumination patterns;
reconstructing the scenes from the captured images with reconstruction parameters;
estimating the reconstruction error and a gradient of the reconstruction error with respect to the control vectors and the reconstruction parameters;
iteratively performing until the reconstruction error reaches a predetermined error condition:
determining a current set of control vectors and current set of reconstruction parameters by updating at least one of the set of control vectors and the set of reconstruction parameters to reduce the reconstruction error;
illuminating the one or more scenes with the illumination patterns governed by the current set of control vectors;
capturing one or more images of each of the scenes while the scene is being illuminated with at least one of the illumination patterns;
reconstructing the scenes from the one or more captured images using the current reconstruction parameters; and
estimating an updated reconstruction error and an updated gradient of the reconstruction error with respect to the current control vectors and the current reconstruction parameters; and
outputting at least one of the current control vectors and the current reconstruction parameters.

### Claim 2 (depends on 1)

The method of claim 1, wherein estimating the reconstruction error comprises evaluating a function that penalizes depth errors with respect to a ground truth, and wherein iteratively reducing the reconstruction error comprises performing at least one of stochastic gradient descent and derivative-free optimization.

### Claim 3 (depends on 1)

The method of claim 1, wherein the initial control vectors comprise at least one of pre-existing control vectors, random control vectors, or low-contrast random control vectors.

### Claim 4 (depends on 1)

The method of claim 1, wherein updating the set of control vectors also comprises incorporating user-defined constraints comprising at least one of frequency content of the illumination patterns, amplitude of the illumination patterns, and total energy consumption of the illumination patterns.

### Claim 5 (depends on 1)

The method of claim 1, wherein the one or more scenes are computationally generated and restricted to lie in a selected subset of 3D space, wherein illuminating the one or more scenes with the illumination pattern comprises a computational simulation, wherein capturing the one or more images comprises computationally simulating image formation, and wherein estimating the gradient of the reconstruction error comprises determining a derivative based on an image formation model.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more scenes comprise at least one surface, illuminating the one or more scenes with the illumination patterns comprises optical illumination, capturing the one or more images comprises optically capturing the one or more images, and estimating the gradient of the reconstruction error comprises optically estimating an image Jacobian with respect to the control vectors.

### Claim 7 (depends on 6)

The method of claim 6, wherein the one or more scenes comprise a randomly-textured surface that exhibits at least one of direct surface reflection, sub-surface scattering, or surface inter-reflection.

### Claim 8 (depends on 1)

The method of claim 1, wherein the control vectors comprise at least one of a discretized time-varying illumination pattern and a discretized time-varying pixel demodulation function.

### Claim 9

A system for optimizing depth imaging, the system comprising one or more processors in communication with a data storage, the one or more processors configurable to execute:
an illumination module to direct illumination of one or more scenes with illumination patterns governed by an initial set of control vectors;
a capture module to receive one or more captured images of each of the scenes while the scene is being illuminated with at least one of the illumination patterns;
a reconstruction module to:
reconstruct the scenes from the captured images with reconstruction parameters;
estimate the reconstruction error and a gradient of the reconstruction error with respect to the control vectors and the reconstruction parameters; and
iteratively perform until the reconstruction error reaches a predetermined error condition:
determining a current set of control vectors and current set of reconstruction parameters by updating at least one of the set of control vectors and the set of reconstruction parameters to reduce the reconstruction error;
illuminating the one or more scenes with the illumination patterns governed by the current set of control vectors;
capturing one or more images of each of the scenes while the scene is being illuminated with at least one of the illumination patterns;
reconstructing the scenes from the one or more captured images using the current reconstruction parameters; and
estimating an updated reconstruction error and an updated gradient of the reconstruction error with respect to the current control vectors and the current reconstruction parameters; and
an output interface to output at least one of the updated control vectors and the updated reconstruction parameters.

### Claim 10 (depends on 9)

The system of claim 9, wherein estimating the reconstruction error comprises evaluating a function that penalizes depth errors with respect to a ground truth, and wherein iteratively reducing the reconstruction error comprises performing at least one of stochastic gradient descent and derivative-free optimization.

### Claim 11 (depends on 9)

The system of claim 9, wherein the initial control vectors comprise at least one of pre-existing control vectors, random control vectors, or low-contrast random control vectors.

### Claim 12 (depends on 9)

The system of claim 9, wherein updating the set of control vectors also comprises incorporating user-defined constraints comprising at least one of frequency content of the illumination patterns, amplitude of the illumination patterns, and total energy consumption of the illumination patterns.

### Claim 13 (depends on 9)

The system of claim 9, wherein the one or more scenes are computationally generated and restricted to lie in a selected subset of 3D space, wherein illuminating the one or more scenes with the illumination pattern comprises a computational simulation, wherein capturing the one or more images comprises computationally simulating image formation, and wherein estimating the gradient of the reconstruction error comprises determining a derivative based on an image formation model.

### Claim 14 (depends on 9)

The system of claim 9, wherein the one or more scenes comprise at least one surface, illuminating the one or more scenes with the illumination patterns comprises optical illumination, capturing the one or more images comprises optically capturing the one or more images, and estimating the gradient of the reconstruction error comprises optically estimating an image Jacobian with respect to the control vectors.

### Claim 15

A computer-implemented method for generating a depth image of a scene, the method comprising:
illuminating the scene with one or more illumination patterns, each pattern comprising a plurality of discretized elements, intensity of each element governed by a code vector;
capturing one or more images of the scene while the scene is being illuminated;
for each pixel, generating an observation vector comprising at least one intensity recorded at the pixel for each of the captured images;
for each pixel, determining the code vector that best corresponds with the respective observation vector by maximizing the zero-mean normalized cross-correlation (ZNCC);
for each pixel, determining a depth value from the best-corresponding code vector; and
outputting the depth values as a depth image.

### Claim 16 (depends on 9)

The method of claim 9, wherein each observation vector incorporates intensities of neighbouring image pixels, and wherein each code vector incorporates neighbouring discretized intensities.

### Claim 17 (depends on 10)

The method of claim 10, further comprising:
using a trained artificial neural network to transform each observation vector to a higher-dimensional feature vector; and
using a trained artificial neural network to transform each code vector to a higher-dimensional feature vector,
wherein determining the code vector that best corresponds with the respective observation vector comprises maximizing the ZNCC between the transformed respective observation vector and the transformed code vectors.

### Claim 18 (depends on 9)

The method of claim 9, wherein each illumination pattern is a discretized two-dimensional pattern that is projected onto a scene from a viewpoint that is distinct from the captured images, wherein each element in the pattern is a projected pixel, and wherein determining the depth value from the best-corresponding code vector comprises triangulation.

### Claim 19 (depends on 9)

The method of claim 9, wherein each illumination pattern comprises multiple wavelength bands, wherein the observation vector at each pixel comprises the raw or demosaiced intensities of each wavelength band for the respective pixel.

### Claim 20 (depends on 9)

The method of claim 9, wherein the discretized elements of each illumination pattern comprise a discretized time-varying pattern that modulates the intensity of a light source, each element in the pattern is associated with a time-of-flight delay and a code vector, and wherein determining the depth value from the best-corresponding code vector comprises multiplication by the speed of light.
