# US 10,767,981 B2 · Systems and methods for estimating depth from projected texture using camera arrays

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,767,981 B2
- Document ID: US10767981B2
- Status: Granted
- Kind: B2
- Application number: 16/177,191
- Filed: 2018-10-31
- Published: 2019-02-28
- Granted: 2020-09-08
- Earliest claimed: 2013-11-18
- Assignees: FotoNation Limited
- Applicants: FotoNation Limited
- Inventors: Kartik Venkataraman; Jacques Duparré
- Web page: https://patentgrep.com/patent/US10767981B2

## CPC

- G01B 11/22: for measuring depth
- G01B 11/2545: {with one projection direction and several detection directions, e.g. stereo}
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/557: from light fields, e.g. from plenoptic cameras
- G06T 7/596: {from three or more stereo images}
- G06T 2207/10024: Color image
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods in accordance with embodiments of the invention estimate depth from projected texture using camera arrays. One embodiment of the invention includes: at least one two-dimensional array of cameras comprising a plurality of cameras; an illumination system configured to illuminate a scene with a projected texture; a processor; and memory containing an image processing pipeline application and an illumination system controller application. In addition, the illumination system controller application directs the processor to control the illumination system to illuminate a scene with a projected texture. Furthermore, the image processing pipeline application directs the processor to: utilize the illumination system controller application to control the illumination system to illuminate a scene with a projected texture capture a set of images of the scene illuminated with the projected texture; determining depth estimates for pixel locations in an image from a reference viewpoint using at least a subset of the set of images. Also, generating a depth estimate for a given pixel location in the image from the reference viewpoint includes: identifying pixels in the at least a subset of the set of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths along a plurality of epipolar lines aligned at different angles; comparing the similarity of the corresponding pixels identified at each of the plurality of depths; and selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint.

## Claims

### Claim 1

A camera array, comprising:
at least two two-dimensional arrays of cameras comprising a first two-dimensional array of cameras and a second two-dimensional array of cameras, each comprising a plurality of cameras, wherein a horizontal baseline between the at least two two-dimensional arrays of cameras is larger than a vertical baseline between cameras within each array of cameras;
an illumination system configured to illuminate a scene with a projected texture;
a processor;
memory containing an image processing pipeline application and an illumination system controller application;
wherein the illumination system controller application directs the processor to control the illumination system to illuminate a scene with a projected texture;
wherein the image processing pipeline application directs the processor to:
utilize the illumination system controller application to control the illumination system to illuminate a scene with a projected texture;
capture a set of images of the scene illuminated with the projected texture;
determining depth estimates for pixel locations in an image from a reference viewpoint using at least a subset of the set of images, wherein generating a depth estimate for a given pixel location in the image from the reference viewpoint comprises:
identifying pixels in the at least a subset of the set of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths along a plurality of epipolar lines aligned at different angles, wherein disparity observed along a first epipolar line corresponding to a horizontal baseline between a camera in the first two-dimensional array of cameras and a camera in the second two-dimensional array of cameras is greater than disparity observed along a second perpendicular epipolar line between at least two cameras within at least one of the two-dimensional arrays of cameras;
comparing the similarity of the corresponding pixels identified at each of the plurality of depths; and
selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint.

### Claim 2 (depends on 1)

The camera array of claim 1, wherein the at least two two-dimensional arrays of cameras are located in complementary occlusion zones surrounding the illumination system.

### Claim 3 (depends on 2)

The camera array of claim 2, wherein a portion of a scene that is occluded in the field of view of at least one camera in a first of the two-dimensional arrays of cameras is visible in a plurality of cameras in a second of the arrays of cameras, where the first and second arrays of cameras are located in complementary occlusion zones on opposite sides of the illumination system.

### Claim 4 (depends on 3)

The camera array of claim 3, wherein the at least two two-dimensional arrays of cameras comprises a pair of two-dimensional arrays of cameras located in complementary occlusion zones on either side of the illumination system.

### Claim 5 (depends on 4)

The camera array of claim 4, wherein each array of cameras is a 2×2 array of monochrome cameras.

### Claim 6 (depends on 4)

The camera array of claim 4, wherein the projected texture includes a first spatial pattern period in a first direction and a second larger spatial pattern period in a second direction.

### Claim 7 (depends on 1)

The camera array of claim 1, wherein the at least two two-dimensional array of cameras comprises one two-dimensional array of cameras including a plurality of lower resolution cameras and at least one higher resolution camera.

### Claim 8 (depends on 7)

The camera array of claim 7, wherein the two-dimensional array of cameras comprises at least one lower resolution camera located above, below, to the left, and to the right of the higher resolution camera.

### Claim 9 (depends on 8)

The camera array of claim 8, wherein the higher resolution camera includes a Bayer filter pattern and the lower resolution cameras are monochrome cameras.

### Claim 10 (depends on 9)

The camera array of claim 9, wherein the image processing pipeline application configures the higher resolution camera to capture texture information when the illumination system is not illuminating the scene using the projected pattern.

### Claim 11 (depends on 1)

The camera array of claim 1, wherein the projected texture includes a first spatial pattern period in a first direction and a second larger spatial pattern period in a second direction.

### Claim 12 (depends on 1)

The camera array of claim 1, wherein the illumination system is a static illumination system configured to project a fixed pattern.

### Claim 13 (depends on 1)

The camera array of claim 1, wherein:
the illumination system is a dynamic illumination system configured to project a controllable pattern; and
the illumination system controller application directs the processor to control the pattern projected by the illumination system.

### Claim 14 (depends on 1)

The camera array of claim 1, wherein the illumination system includes a spatial light modulator selected from the group consisting of a reflective liquid crystal on silicon microdisplay and a translucent liquid crystal microdisplay.

### Claim 15 (depends on 14)

The camera array of claim 14, wherein the image processing pipeline application directs the processor to:
utilize the illumination system controller application to control the illumination system to illuminate a scene with a first projected texture;
capture a first set of images of the scene illuminated with the first projected texture;
determine initial depth estimates for pixel locations in an image from a reference viewpoint using at least a subset of the first set of images;
utilize the illumination system controller application to control the illumination system to illuminate a scene with a second projected texture selected based upon at least one initial depth estimate for a pixel location in an image from a reference viewpoint;
capture a second set of images of the scene illuminated with the second projected texture; and
determine updated depth estimates for pixel locations in an image from a reference viewpoint using at least a subset of the first set of images.

### Claim 16 (depends on 15)

The camera array of claim 15, wherein the spatial pattern period of the second projected texture at the at least one initial depth estimate for a pixel location in an image from a reference viewpoint is higher than the spatial resolution of the plurality of cameras at the at least one initial depth estimate for a pixel location in an image from the reference viewpoint.

### Claim 17 (depends on 1)

The camera array of claim 1, wherein the illumination system comprises an array of projectors.

### Claim 18 (depends on 17)

The camera array of claim 17, wherein the array of projectors comprises projectors configured to project different patterns.

### Claim 19 (depends on 18)

The camera array of claim 18, wherein the different patterns comprise patterns having different spatial pattern periods.

### Claim 20 (depends on 18)

The camera array of claim 18, wherein:
the projectors are configured to project controllable patterns; and
the illumination system controller application directs the processor to control the patterns projected by the illumination system.