US 9,872,011 B2 · High-speed depth sensing with a hybrid camera setup

Abstract

In a method, a first video stream is received from a video camera, and a second video stream is received from a depth camera. A pixel mapping between the video camera and the depth camera is known. The video camera has an update rate greater than that of the depth camera. Optical flow in successive frames of the first video stream is measured, and a portion of the optical flow attributable to depth change is extracted. A scaling factor is calculated for each pixel in successive frames of the first video stream to determine whether a depth change has occurred. A perspective depth correction is applied to each pixel having a depth change. The perspective depth correction is based upon the depth of the corresponding pixel in the most recent frame from the second video stream. A combined video stream having an update rate of the video camera and depth information from the depth camera is obtained.

Claims22

Dependent claims

An apparatus comprising:

a depth camera;
a video camera, said video camera having an update rate greater than that of said depth camera;
at least one processor;
and
at least one memory including computer program code, the at least one memory and the computer program code configured, with the at least one processor, to cause the apparatus to perform:
receiving a first video stream from the video camera;
receiving a second video stream from the depth camera, wherein a pixel mapping between the video camera and the depth camera is known;
measuring optical flow in successive frames of said first video stream;
extracting a portion of the optical flow attributable to depth change;
calculating a scaling factor for each pixel in successive frames of said first video stream to determine whether a depth change has occurred;
applying a perspective depth correction to each pixel having a depth change, said perspective depth correction being based upon the depth of the corresponding pixel in the most recent frame from the second video stream;
and
outputting a combined video stream having an update rate of the video camera and depth information from said depth camera.

10 more elements

A computer program product comprising a non-transitory computer-readable storage medium bearing computer program code embodied therein for use with an apparatus, the computer program code comprising code for performing at least the following:

receiving a first video stream from a video camera;
receiving a second video stream from a depth camera, wherein a pixel mapping between the video camera and the depth camera is known, and wherein said video camera has an update rate greater than that of said depth camera;
measuring optical flow in successive frames of said first video stream;
extracting a portion of the optical flow attributable to depth change;
calculating a scaling factor for each pixel in successive frames of said first video stream to determine whether a depth change has occurred;
applying a perspective depth correction to each pixel having a depth change, said perspective depth correction being based upon the depth of the corresponding pixel in the most recent frame from the second video stream;
and
outputting a combined video stream having an update rate of the video camera and depth information from said depth camera.

5 more elements

Dependencies are inferred from the claim text.

High-speed depth sensing with a hybrid camera setup

Life of this patent

High-speed depth sensing with a hybrid camera setup

Life of this patent

Abstract

Claims22