# US 10,311,282 B2 · Depth from time of flight camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,282 B2
- Document ID: US10311282B2
- Status: Granted
- Kind: B2
- Application number: 15/701,170
- Filed: 2017-09-11
- Published: 2017-12-28
- Granted: 2019-06-04
- Earliest claimed: 2014-10-14
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Jamie Daniel Joseph Shotton; Cem Keskin; Christoph Rhemann; Toby Sharp; Duncan Paul Robertson; Pushmeet Kohli; Andrew William Fitzgibbon; Shahram Izadi
- Web page: https://patentgrep.com/patent/US10311282B2

## CPC

- G01S 17/36: with phase comparison between the received signal and the contemporaneously transmitted signal
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 7/4911: Transmitters
- G01S 17/10: using transmission of interrupted, pulse-modulated waves (determination of distance by phase measurements G01S17/32)
- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G06F 18/24323: {Tree-organised classifiers}
- G06T 7/11: Region-based segmentation
- G06T 7/50: Depth or shape recovery
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/20: in augmented reality scenes
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10048: Infrared image
- G06T 2207/10152: Varying illumination
- G06T 2207/20081: Training; Learning
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Region of interest detection in raw time of flight images is described. For example, a computing device receives at least one raw image captured for a single frame by a time of flight camera. The raw image depicts one or more objects in an environment of the time of flight camera (such as human hands, bodies or any other objects). The raw image is input to a trained region detector and in response one or more regions of interest in the raw image are received. A received region of interest comprises image elements of the raw image which are predicted to depict at least part of one of the objects. A depth computation logic computes depth from the one or more regions of interest of the raw image.

## Claims

### Claim 1

A method comprising:
receiving at least one image captured for a single frame by a time of flight camera, the image depicting one or more objects in an environment of the time of flight camera;
inputting the at least one image into a trained region detector;
determining, by the trained region detector, one or more regions of interest in the at least one image based on associations between pixels of time of flight images and regions that depict an object of interest in the time of flight images,
wherein a region of interest comprises image elements which are predicted to depict at least part of one of the objects; and
computing depth from the one or more regions of interest of the at least one image.

### Claim 2 (depends on 1)

The method as claimed in claim 1, wherein the single frame comprises a plurality of sensor data images.

### Claim 3 (depends on 1)

The method as claimed in claim 1 comprising computing depth only from the one or more regions of interest of the at least one image.

### Claim 4 (depends on 1)

The method as claimed in claim 1 wherein the at least one image from the time of flight camera is a plurality of infra-red intensity values.

### Claim 5 (depends on 1)

The method as claimed in claim 1 comprising:
receiving a plurality of images captured for a single frame by the time of flight camera;
aggregating the plurality of images to produce an aggregated image; and
inputting the aggregated image to the trained region detector.

### Claim 6 (depends on 1)

The method as claimed in claim 1 comprising:
receiving a plurality of images captured for a single frame by the time of flight camera; and
inputting each of the images to the trained region detector.

### Claim 7 (depends on 1)

The method as claimed in claim 1 comprising using the trained region detector to classify image elements of the at least one image as being in a region of interest or not.

### Claim 8 (depends on 7)

The method as claimed in claim 7 comprising clustering the classified image elements.

### Claim 9 (depends on 7)

The method as claimed in claim 7 comprising applying a sliding window detector to the classified image elements.

### Claim 10 (depends on 1)

The method as claimed in claim 1 comprising training the region detector using training data comprising time of flight images depicting examples of the object, the time of flight images having labeled regions of interest.

### Claim 11 (depends on 1)

The method as claimed in claim 1 wherein the trained region detector is a random decision forest.

### Claim 12

A computing system comprising:
an input/output controller arranged to receive at least one image captured for a single frame by a time of flight camera, the at least one image depicting one or more objects in an environment of the time of flight camera;
a region detector arranged to compute from the at least one image one or more regions of interest in the at least one image based on associations between pixels of time of flight images and regions that depict an object of interest in the time of flight images, a region of interest comprising image elements which are predicted to depict at least part of one of the objects; and
a depth map computation logic arranged to compute depth from the one or more regions of interest of the at least one image.

### Claim 13 (depends on 12)

The computing system of claim 12, wherein the single frame comprises a plurality of sensor data images.

### Claim 14 (depends on 13)

The computing system of claim 13 the region detector comprising a clustering component arranged to cluster labeled image elements output by the classifier.

### Claim 15 (depends on 13)

The computing system of claim 13 the region detector comprising a sliding window component arranged to compute one or more regions from labeled image elements output by the classifier.

### Claim 16 (depends on 13)

The computing system of claim 13 integrated in a time of flight camera.

### Claim 17

A computer storage media storing instructions which when executed by a computing device control the device to:
receive at least one image captured for a single frame by a time of flight camera, the at least one image depicting one or more objects in an environment of the time of flight camera;
send the at least one image to a region detector, having been trained to learn associations between image elements of time of flight images and regions of interest;
determine, by the trained region detector, one or more regions of interest in the at least one image based on associations between pixels of time of flight images and regions that depict an object of interest in the time of flight images, a region of interest comprising image elements which are predicted to depict at least part of one of the objects; and
computer depth from the one or more regions of interest of the at least one image.

### Claim 18 (depends on 17)

The computer readable medium of claim 17 storing instructions which when executed by a computing device control the device to: receive a plurality of images captured for a single frame by the time of flight camera; aggregate the plurality of images to produce an aggregated image; and input the aggregated image to the region detector.

### Claim 19 (depends on 17)

The computer readable medium of claim 17 storing instructions which when executed by a computing device control the device to receive a plurality of images captured for a single frame by the time of flight camera; and
input each of the plurality of images to the region detector.

### Claim 20 (depends on 17)

The computer readable medium of claim 17 storing instructions which when executed by a computing device control the device to compute depth only from the one or more regions of interest.