# US 11,029,149 B2 · Multipath mitigation for time of flight system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,029,149 B2
- Document ID: US11029149B2
- Status: Granted
- Kind: B2
- Application number: 16/262,440
- Filed: 2019-01-30
- Published: 2020-07-30
- Granted: 2021-06-08
- Earliest claimed: 2019-01-30
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Sergio Ortiz Egea; Hung-Ming Lin
- Web page: https://patentgrep.com/patent/US11029149B2

## CPC

- G01C 3/08: Use of electric radiation detectors
- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G01S 7/493: Extracting wanted echo signals
- G06T 5/20: using local operators
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- H04N 23/56: provided with illuminating means
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10152: Varying illumination
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A time-of-flight (ToF) system disclosed herein provides a method of a method of separating a direct component of light collected by a time of flight (ToF) detector from a global component of light collected by the ToF detector, the method comprising acquiring three or more images represented by three or more matrices in response to illuminating a target with a light source using a first spatial pattern at three or more different modulation frequencies, acquiring an additional image represented by an additional matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern, and determining one or more parameters of the direct component of light and the global component of light based on analysis of the three or more matrices and the additional matrix.

## Claims

### Claim 1

A physical hardware system to provide multipath mitigation in a time-of-flight (ToF) system, comprising:
a camera configured to acquire k images, k being two or more, represented by k matrices in response to illuminating a target with a light source using a first spatial pattern at k different light frequencies and to acquire an additional k+1th image represented by an additional matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern;
a denoising module configured to generate a denoising coefficient matrix based on a kernel matrix applied to each of the k+1 images; and
an image analyzer to apply the denoising coefficient to the two or more matrices and the additional k+1th matrix and to determine one or more parameters of a direct component of light and a global component of light received at the camera from the target.

### Claim 2 (depends on 1)

The physical hardware system of claim 1, wherein the first spatial pattern is a uniform spatial pattern.

### Claim 3 (depends on 1)

The physical hardware system of claim 1, wherein the second spatial pattern is a non-uniform spatial pattern.

### Claim 4 (depends on 1)

The physical hardware system of claim 1, wherein each of the first spatial pattern and the second spatial pattern is a non-uniform spatial pattern.

### Claim 5 (depends on 1)

The physical hardware system of claim 1, wherein the additional image is acquired in response to illuminating the target with the light source using the second spatial pattern at a light frequency similar to one of the two or more different light frequencies used when illuminating the target with the first spatial pattern.

### Claim 6 (depends on 1)

The physical hardware system of claim 1, wherein the camera is further configured to acquire three or more images represented by three or more matrices in response to illuminating the target with the light source using the first spatial pattern at three or more different light frequencies.

### Claim 7 (depends on 1)

The physical hardware system of claim 1, wherein the non-uniform spatial pattern is at least one of a dot-pattern, a vertical-line pattern, and a horizontal line pattern.

### Claim 8 (depends on 1)

The physical hardware system of claim 1, wherein the kernel matrix is an m×n matrix with m being different than n.

### Claim 9

A method of separating a direct component of light collected by a time of flight (ToF) detector from a global component of light collected by the ToF detector, the method comprising:
acquiring k images, k being two or more, represented by k matrices in response to illuminating a target with a light source using a first spatial pattern at k different modulation frequencies;
acquiring an additional k+1th image represented by an additional matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern;
generate a denoising coefficient matrix based on a kernel matrix applied to each of the k+1 images; and
applying the denoising coefficient to the two or more matrices and the additional k+1th matrix and to determine one or more parameters of the direct component of light and the global component of light based on analysis of the three or more matrices and the additional matrix.

### Claim 10 (depends on 9)

The method of claim 9, wherein the first spatial pattern is a uniform spatial pattern.

### Claim 11 (depends on 9)

The method of claim 9, wherein the second spatial pattern is a non-uniform spatial pattern.

### Claim 12 (depends on 9)

The method of claim 9, wherein each of the first spatial pattern and the second spatial pattern is a non-uniform spatial pattern.

### Claim 13 (depends on 9)

The method of claim 9, wherein the additional image is acquired in response to illuminating the target with the light source using the second spatial pattern at a light frequency similar to a one of the three or more different light frequencies used when illuminating the target with the first spatial pattern.

### Claim 14 (depends on 9)

The method of claim 9, wherein the non-uniform spatial pattern is at least one of a dot-pattern, a vertical-line pattern, and a horizontal line pattern.

### Claim 15 (depends on 9)

The method of claim 9, wherein the kernel matrix is an m×n matrix with m being different than n.

### Claim 16

A physical article of manufacture including one or more non-transitory computer-readable storage media, encoding computer-executable instructions for executing on a computer system a computer process, the computer process comprising:
acquiring k images, k being two or more, represented by k matrices in response to illuminating a target with a light source using a first spatial pattern at k different modulation frequencies;
acquiring an additional k+1th image represented by an additional matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern;
generating a denoising coefficient matrix based on a kernel matrix applied to each of the k+1 images; and
applying the denoising coefficient to the two or more matrices and the additional k+1th matrix and to determine one or more parameters of the direct component of light and the global component of light based on analysis of the three or more matrices and the additional matrix.

### Claim 17 (depends on 16)

The physical article of manufacture of claim 16, wherein the first spatial pattern is a uniform spatial pattern and the second spatial pattern is a non-uniform spatial pattern.

### Claim 18 (depends on 17)

The physical article of manufacture of claim 17, wherein the process further comprising performing a frequency conversion operation to transform a phase into a frequency K.

### Claim 19 (depends on 17)

The physical article of manufacture of claim 17, wherein each of the first spatial pattern and the second spatial pattern is a non-uniform spatial pattern.

### Claim 20 (depends on 19)

The physical article of manufacture of claim 19, wherein the light source illuminates a target using N temporal modulation frequencies with the uniform spatial pattern and N temporal modulation frequencies with the non-uniform spatial pattern.