# US 2016/0239725 A1 · Method for Denoising Time-of-Flight Range Images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0239725 A1
- Document ID: US20160239725A1
- Status: Published application
- Kind: A1
- Application number: 14/620,610
- Filed: 2015-02-12
- Published: 2016-08-18
- Earliest claimed: 2015-02-12
- Assignees: Mitsubishi Electric Research Laboratories, Inc.
- Applicants: Mitsubishi Electric Research Laboratories, Inc.
- Inventors: Ming-Yu Liu; Yuichi Taguchi; Kilho Son
- Web page: https://patentgrep.com/patent/US20160239725A1

## CPC

- G06T 5/70: Denoising; Smoothing
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 7/13: Edge detection
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/60: Analysis of geometric attributes
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20004: Adaptive image processing
- G06T 2207/20024: Filtering details
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- G06T 2207/20192: Edge enhancement; Edge preservation
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for denoising a range image acquired by a time-of-flight (ToF) camera by first determining locations of edges, and a confidence value of each pixel, and based on the locations of the edges, determining geodesic distances of neighboring pixels. Based on the confidence values, reliabilities of the neighboring pixels are determined and scene dependent noise is reduced using a filter.

## Claims

### Claim 1

A method for denoising an image, wherein the image is a time-of-flight (ToF) range image of pixels, comprising steps of:
determining locations of edges, and a confidence value of each pixel;
determining, based on the locations of the edges, geodesic distances of neighboring pixels;
determining, based on the confidence values, reliabilities of the neighboring pixels; and
reducing scene dependent noise using a filter, wherein the ToF range image is acquired by a ToF range camera, and the steps are performed in a processor.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
calibrating the range image, wherein the calibrating further comprises:
reducing scene independent noise using a pixel-wise calibration using a checkboard calibration pattern, wherein intensities of the checkboard calibration pattern pattern gradually increase from 0% to 50% for darker squares, and from 50% to 100% for lighter squares.

### Claim 3 (depends on 1)

The method of claim 1, wherein the locations of the edges are determined using a neural network.

### Claim 4 (depends on 1)

The method of claim 1, wherein the confidence values are determined using a neural network.

### Claim 5 (depends on 1)

The method of claim 1, wherein the filter is
R
^

(
p
)
=
∑
q
∈
N

(
p
)

w

(
p
,
q
)

R

(
q
)
∑
q
∈
N

(
p
)

w

(
p
,
q
)
,
(
1
)
where R(p) is a range at pixel p in the input range image, {circumflex over (R)}(p) is the range value at p N(p) is the neighboring pixels of p, and w(p, q) is a weighting function measuring a relation between the pixels p and q.

### Claim 6 (depends on 5)

The method of claim 5, wherein the weighting function is assymmetric and has two exponnential (exp) components
w

(
p
,
q
)
=
exp

(
γ


C
2

(
q
)
)

exp
(
-
G
2

(
p
,
q
)
2

σ
2
)
,
(
2
)
where a first factor measures the confidence of pixel q and a second factor measures a similarity of p and q, γ and σ represent a shape of the filter, G represents the geodesic distance.

### Claim 7 (depends on 1)

The method of claim 1, wherein the ToF range camera is attached on a robot arm and moved to different viewpoints to acquire a plane dataset.

### Claim 8 (depends on 3)

The method of claim 3, wherein the neural network is learned from a scene dataset including pairs of ToF range images and ground truth range images.

### Claim 9 (depends on 4)

The method of claim 4, wherein the neural network is learned from a scene dataset including pairs of ToF range images and ground truth range images.

### Claim 10 (depends on 9)

The method of claim 9, wherein the ground truth range images are acquired by a structured light sensor.

### Claim 11 (depends on 10)

The method of claim 10, wherein the structured light sensor and the ToF range camera are attached on a robot arm and moved to different viewpoints to acquire the scene dataset.
