# US 2021/0333396 A1 · 3d sensing depth camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0333396 A1
- Document ID: US20210333396A1
- Status: Published application
- Kind: A1
- Application number: 17/238,521
- Filed: 2021-04-23
- Published: 2021-10-28
- Earliest claimed: 2020-04-23
- Assignees: Duke University
- Applicants: Duke University
- Inventors: Joseph IZATT; Ruobing QIAN; Kevin ZHOU
- Web page: https://patentgrep.com/patent/US20210333396A1

## CPC

- G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46)
- G02B 26/105: {with one or more pivoting mirrors or galvano-mirrors (G02B26/101 takes precedence)}
- G01S 7/4817: {relating to scanning}
- G01S 17/89: for mapping or imaging
- G02B 5/1866: {Transmission gratings characterised by their structure, e.g. step profile, contours of substrate or grooves, pitch variations, materials (G02B5/1809, G02B5/1828, G02B5/1833, G02B5/1838 and G02B5/1847 take precedence)}
- G02B 26/106: {having diffraction gratings as scanning elements, e.g. holographic scanners (holographic optical elements G02B5/32, holography G03H)}
- G02B 27/30: Collimators
- G06T 5/20: using local operators
- G06T 5/70: Denoising; Smoothing
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20032: Median filtering
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A 3D sensing depth camera involves a time-frequency multiplexed frequency-modulated continuous wave (FMCW) LiDAR technique. A sample arm of such a system for 3D depth sensing includes a scanner for beam scanning in a first axis and a diffractive optical element for spectrally encoded scanning along a second axis. The sample arm can further include beam shaping optics such as a collimator and a lens. Processing of an interferogram comprising depth information at a different position along the second axis for each frequency sweep of the light source involves applying a windowed spectral estimator at a particular spectral window size with zero-padding and according to a specified lateral sampling approach.

## Claims

### Claim 1

A system for 3D depth sensing:
a light source that is swept across frequencies from a first frequency to a last frequency;
a beam splitter system that splits the light source into at least a reference beam and a sample beam;
a detector that collects an interference signal between light returning from the reference beam and the sample beam, the interference signal forming an interferogram;
a sample arm comprising a scanner for beam scanning in a first axis and a diffractive optical element for spectrally encoded scanning along a second axis, wherein for each spectral window of the light source, the interferogram comprises depth information at a different position along the second axis; and
a processing system coupled to the detector, the processing system comprising a hardware processor and a memory storing instructions that when executed by the hardware processor direct the processing system to:
receive the interferogram of each frequency sweep of the light source from the detector;
for each interferogram, apply a windowed spectral estimator at a particular spectral window size with zero-padding and according to a specified lateral sampling approach by:
sampling the interferogram according to the specified lateral sampling approach; and
for each spectral window sample of the interferogram, applying zero-padding to a particular number of samples and applying a spectral estimator to the zero-padded spectral window sample to generate a signal for a pixel; performing peak localization analysis on the signal to identify a dominant peak; and if the dominant peak is identified, assigning a depth value according to a corresponding depth position of the dominant peak for the pixel, otherwise, assigning the pixel as a background; and
provide a depth map of the pixels comprising the assigned depth values.

### Claim 2 (depends on 1)

The system of claim 1, wherein the sample arm further comprises sample beam shaping optics.

### Claim 3 (depends on 2)

The system of claim 2, wherein the sample beam shaping optics and diffractive optical element are configured to direct the light comprising each spectral window of the light source on or within a diameter of single Airy disk.

### Claim 4 (depends on 2)

The system of claim 2, wherein the sample beam shaping optics comprise:
a collimator that directs the sample beam to the scanner; and
a lens to focus the sample beam.

### Claim 5 (depends on 1)

The system of claim 1, wherein the specified lateral sampling approach comprises a Nyquist lateral sampling.

### Claim 6 (depends on 1)

The system of claim 1, wherein the spectral window size is based on a lateral resolution of the system such that if a point spread function of two adjacent wavelengths are unresolvable, detected signals from the two adjacent wavelengths are treated as arising from a same position and are analyzed within a same spectral window.

### Claim 7 (depends on 1)

The system of claim 1, wherein the spectral window size is between 0.01 nm and 1.0 nm in wavelength, a lateral resolution along the second axis is between 0.1 mm and 10 mm and a lateral resolution along the first axis is between 0.1 mm and 10 mm.

### Claim 8 (depends on 1)

The system of claim 1, wherein each spectral window sample includes a truncated signal of the interferogram, wherein the particular number of samples for zero-padding is 1000-10,000 samples.

### Claim 9 (depends on 1)

The system of claim 1, wherein the windowed spectral estimator is a short-time Fourier transform.

### Claim 10 (depends on 1)

The system of claim 1, wherein the dominant peak is identified as a highest peak above a predetermined threshold.

### Claim 11

A sample arm for 3D depth sensing, comprising:
a collimator for collimating a received sample beam from a light source,
a scanner for beam scanning of the sample beam in a first axis,
a lens for collimating or focusing the sample beam, and
a diffractive optical element for spectrally encoded scanning along a second axis.

### Claim 12 (depends on 11)

The sample arm of claim 11, wherein the collimator, lens, and diffractive optical element are configured to direct light comprising each spectral window of the light source on or within a diameter of single Airy disk.

### Claim 13 (depends on 11)

The sample arm of claim 11, wherein the diffractive optical element comprises a transmissive grating.

### Claim 14 (depends on 11)

The sample arm of claim 11, wherein the collimator is a reflective collimator.

### Claim 15 (depends on 11)

The sample arm of claim 11, wherein the lens is disposed between the scanner and the diffractive optical element.

### Claim 16

A method of generating a depth map, comprising:
receiving an interferogram;
sampling the interferogram according to a specified lateral sampling approach to obtain a spectral window sample; and
for each spectral window sample of the interferogram:
applying zero-padding to a particular number of samples,
applying a spectral estimator to the zero-padded spectral window sample to generate a signal for a pixel;
performing peak localization analysis on the signal to identify a dominant peak; and
if the dominant peak is identified, assigning a depth value according to a corresponding depth position of the dominant peak for the pixel, otherwise, assigning the pixel as a background.

### Claim 17 (depends on 16)

The method of claim 16, wherein the spectral estimator is a Fourier transform.

### Claim 18 (depends on 16)

The method of claim 16, wherein a spectral window size of the spectral window is based on a lateral resolution of a system providing the interferogram such that if a point spread function of two adjacent wavelengths are unresolvable, detected signals from the two adjacent wavelengths are treated as arising from a same position and are analyzed within a same spectral window.

### Claim 19 (depends on 16)

The method of claim 16, wherein each spectral window sample includes a truncated signal of the interferogram, wherein the particular number of samples for zero-padding is 1000-10,000 samples.

### Claim 20 (depends on 16)

The method of claim 16, further comprising applying a gradient-based background noise removal algorithm and median filter to a depth map of the pixels comprising the assigned depth values.