# US 2022/0036571 A1 · Dynamic structured light for depth sensing systems based on contrast in a local area

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036571 A1
- Document ID: US20220036571A1
- Status: Published application
- Kind: A1
- Application number: 17/504,004
- Filed: 2021-10-18
- Published: 2022-02-03
- Earliest claimed: 2018-05-21
- Assignees: Facebook Technologies, LLC
- Applicants: Facebook Technologies, LLC
- Inventors: Michael Hall; Xinqiao Liu; Zhaoming Zhu; Rajesh Lachhmandas Chhabria; Huixuan Tang; Shuochen Su; Zihe Gao
- Web page: https://patentgrep.com/patent/US20220036571A1

## CPC

- G01B 11/2513: {with several lines being projected in more than one direction, e.g. grids, patterns}
- G01B 11/22: for measuring depth
- G02B 27/0172: {characterised by optical features}
- G06T 7/11: Region-based segmentation
- G06T 7/174: involving the use of two or more images
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/529: from texture
- G06T 7/55: from multiple images
- G06T 7/593: from stereo images
- G06T 7/90: Determination of colour characteristics
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06V 10/145: Illumination specially adapted for pattern recognition, e.g. using gratings
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597)
- H04N 13/128: Adjusting depth or disparity
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 23/56: provided with illuminating means
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A depth camera assembly (DCA) determines depth information. The DCA projects a dynamic structured light pattern into a local area and captures images including a portion of the dynamic structured light pattern. The DCA determines regions of interest in which it may be beneficial to increase or decrease an amount of texture added to the region of interest using the dynamic structured light pattern. For example, the DCA may identify the regions of interest based on contrast values calculated using a contrast algorithm, or based on the parameters received from a mapping server including a virtual model of the local area. The DCA may selectively increase or decrease an amount of texture added by the dynamic structured light pattern in portions of the local area. By selectively controlling portions of the dynamic structured light pattern, the DCA may decrease power consumption and/or increase the accuracy of depth sensing measurements.

## Claims

### Claim 1

A depth camera assembly (DCA) comprising:
a structured light (SL) projector configured to project one or more SL patterns into a local area in accordance with illumination instructions;
a camera assembly configured to capture images of a portion of the local area including the one or more SL patterns; and
a controller configured to:
provide first illumination instructions to the SL projector, wherein the first illumination instructions cause the SL projector to project a first SL pattern having a first pattern element density into a first region of interest based on a first amount of contrast in the first region of interest; and
provide second illumination instructions to the SL projector, wherein the second illumination instructions cause the SL projector to project a second SL pattern having a second pattern element density into a second region of interest based on a second amount of contrast in the second region of interest, wherein the second amount of contrast is greater than the first amount of contrast.

### Claim 2 (depends on 1)

The DCA of claim 1, wherein the controller is further configured to:
transmit, to a mapping server, a location of the DCA, wherein the mapping server stores a virtual model describing a plurality of physical spaces;
receive, from the mapping server, the first amount of contrast for the first region of interest; and
receive, from the mapping server, the second amount of contrast for the second region of interest.

### Claim 3 (depends on 2)

The DCA of claim 2, wherein the mapping server is configured to determine the first amount of contrast based on a virtual model of the local area.

### Claim 4 (depends on 2)

The DCA of claim 2, wherein the controller is configured to identify the first region of interest and the second region of interest based on parameters received from the mapping server.

### Claim 5 (depends on 1)

The DCA of claim 1, wherein the controller is configured to identify the first region of interest in the local area based on detecting movement of an object within the first region of interest.

### Claim 6 (depends on 1)

The DCA of claim 1, wherein the controller is configured to:
identify the first region of interest in the local area based on detecting an occluded object within the first region of interest; and
decrease an amount of texture in the first region of interest in response to detecting the occluded object, wherein the occluded object is a physical object, and wherein the physical object is occluded by a virtual object.

### Claim 7 (depends on 1)

The DCA of claim 1, wherein the controller is configured to:
determine an expected location of a virtual object within the local area; and
select a power level for the first SL pattern based on the expected location of the virtual object.

### Claim 8 (depends on 1)

The DCA of claim 1, wherein the intensity of the second SL pattern is different than the intensity of the first SL pattern.

### Claim 9 (depends on 1)

The DCA of claim 1, wherein the controller is configured to transmit visual information describing a portion of the local area to a mapping server.

### Claim 10 (depends on 1)

The DCA of claim 1, wherein the controller is configured to:
subtract a first image captured without any SL pattern from a second image containing one of the SL patterns; and
determine, based on the subtraction, a background level of IR illumination.

### Claim 11

A method comprising:
providing first illumination instructions to a structured light (SL) projector, wherein the first illumination instructions cause the SL projector to project a first SL pattern having a first pattern element density into a first region of interest based on a first amount of contrast in the first region of interest; and
providing second illumination instructions to the SL projector, wherein the second illumination instructions cause the SL projector to project a second SL pattern having a second pattern element density into a second region of interest based on a second amount of contrast in the second region of interest, wherein the second amount of contrast is greater than the first amount of contrast.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
transmitting a location of a local area including the first region of interest and the second region of interest to a mapping server, wherein the mapping server stores a virtual model describing a plurality of physical spaces;
receiving, from the mapping server, the first amount of contrast for the first region of interest; and
receiving, from the mapping server, the second amount of contrast for the second region of interest.

### Claim 13 (depends on 12)

The method of claim 12, wherein the mapping server is configured to determine the first amount of contrast based on a virtual model of the local area.

### Claim 14 (depends on 12)

The method of claim 12, further comprising identifying the first region of interest and the second region of interest based on parameters received from the mapping server.

### Claim 15 (depends on 11)

The method of claim 11, further comprising:
identifying the first region of interest in the local area based on detecting movement of an object within the first region of interest.

### Claim 16 (depends on 11)

The method of claim 11, further comprising:
identifying the first region of interest in the local area based on detecting an occluded object within the first region of interest.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
decreasing an amount of texture in the first region of interest in response to detecting the occluded object.

### Claim 18 (depends on 17)

The method of claim 17, wherein the occluded object is a physical object, and wherein the physical object is occluded by a virtual object.

### Claim 19 (depends on 11)

The method of claim 11, further comprising transmitting visual information describing a portion of a local area to a mapping server.

### Claim 20

A non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform operations comprising:
providing first illumination instructions to a structured light (SL) projector, wherein the first illumination instructions cause the SL projector to project a first SL pattern having a first pattern element density into a first region of interest based on a first amount of contrast in the first region of interest; and
providing second illumination instructions to the SL projector, wherein the second illumination instructions cause the SL projector to project a second SL pattern having a second pattern element density into a second region of interest based on a second amount of contrast in the second region of interest, wherein the second amount of contrast is greater than the first amount of contrast.
