# US 9,872,011 B2 · High-speed depth sensing with a hybrid camera setup > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,872,011 B2 - Document ID: US9872011B2 - Status: Granted - Kind: B2 - Application number: 14/950,360 - Filed: 2015-11-24 - Published: 2017-05-25 - Granted: 2018-01-16 - Earliest claimed: 2015-11-24 - Assignees: Nokia Technologies Oy - Applicants: Nokia Technologies Oy - Inventors: Xin Chen; Ruigang Yang - Web page: https://patentgrep.com/patent/US9872011B2 ## CPC - G06T 7/269: using gradient-based methods - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 7/292: Multi-camera tracking - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 11/60: Creating or editing images; Combining images with text - H04N 13/139: Format conversion, e.g. of frame-rate or size - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - G06T 2200/04: involving 3D image data - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - H04N 2213/003: Aspects relating to the "2D+depth" image format - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract In a method, a first video stream is received from a video camera, and a second video stream is received from a depth camera. A pixel mapping between the video camera and the depth camera is known. The video camera has an update rate greater than that of the depth camera. Optical flow in successive frames of the first video stream is measured, and a portion of the optical flow attributable to depth change is extracted. A scaling factor is calculated for each pixel in successive frames of the first video stream to determine whether a depth change has occurred. A perspective depth correction is applied to each pixel having a depth change. The perspective depth correction is based upon the depth of the corresponding pixel in the most recent frame from the second video stream. A combined video stream having an update rate of the video camera and depth information from the depth camera is obtained. ## Claims ### Claim 1 An apparatus comprising: a depth camera; a video camera, said video camera having an update rate greater than that of said depth camera; at least one processor; and at least one memory including computer program code, the at least one memory and the computer program code configured, with the at least one processor, to cause the apparatus to perform: receiving a first video stream from the video camera; receiving a second video stream from the depth camera, wherein a pixel mapping between the video camera and the depth camera is known; measuring optical flow in successive frames of said first video stream; extracting a portion of the optical flow attributable to depth change; calculating a scaling factor for each pixel in successive frames of said first video stream to determine whether a depth change has occurred; applying a perspective depth correction to each pixel having a depth change, said perspective depth correction being based upon the depth of the corresponding pixel in the most recent frame from the second video stream; and outputting a combined video stream having an update rate of the video camera and depth information from said depth camera. ### Claim 2 (depends on 1) The apparatus as claimed in claim 1, wherein the video camera has an update rate of 200 Hz. ### Claim 3 (depends on 1) The apparatus as claimed in claim 1, wherein the depth camera has an update rate of 30 Hz. ### Claim 4 (depends on 1) The apparatus as claimed in claim 1, wherein images from said first video stream and images from said second video stream are aligned either optically or through a remapping process. ### Claim 5 (depends on 1) The apparatus as claimed in claim 1, wherein extracting a portion of the optical flow attributable to depth change comprises: applying Helmholtz-Hodge decomposition to decompose a two-dimensional optical flow into a divergence-free vector field and a divergence field; ignoring the divergence-free vector field; and testing the divergence field to determine whether divergence is caused by the depth change of a rigid object. ### Claim 6 (depends on 5) The apparatus as claimed in claim 5, wherein the testing is carried out by comparing unit structures formed by pixels and four spatial neighbors thereof in successive frames to determine whether the unit structures have changed in either size or orientation. ### Claim 7 (depends on 1) The apparatus as claimed in claim 1, wherein the at least one memory and the computer program code are further configured, with the at least one processor, to provide spatial and temporal neighboring pixels with the same depth correction when they have the same color. ### Claim 8 A method comprising: receiving a first video stream from a video camera; receiving a second video stream from a depth camera, wherein a pixel mapping between the video camera and the depth camera is known, and wherein said video camera has an update rate greater than that of said depth camera; measuring optical flow in successive frames of said first video stream; extracting a portion of the optical flow attributable to depth change; calculating a scaling factor for each pixel in successive frames of said first video stream to determine whether a depth change has occurred; applying a perspective depth correction to each pixel having a depth change, said perspective depth correction being based upon the depth of the corresponding pixel in the most recent frame from the second video stream; and outputting a combined video stream having an update rate of the video camera and depth information from said depth camera. ### Claim 9 (depends on 8) The method as claimed in claim 8, wherein images from said first video stream and images from said second video stream are aligned either optically or through a remapping process. ### Claim 10 (depends on 8) The method as claimed in claim 8, wherein extracting a portion of the optical flow attributable to depth change comprises: applying Helmholtz-Hodge decomposition to decompose a two-dimensional optical flow into a divergence-free vector field and a divergence field; ignoring the divergence-free vector field; and testing the divergence field to determine whether divergence is caused by the depth change of a rigid object. ### Claim 11 (depends on 10) The method as claimed in claim 10, wherein the testing is carried out by comparing unit structures formed by pixels and four spatial neighbors thereof in successive frames to determine whether the unit structures have changed in either size or orientation. ### Claim 12 (depends on 8) The method as claimed in claim 8, further comprising: providing spatial and temporal neighboring pixels with the same depth correction when they have the same color. ### Claim 13 An apparatus comprising: means for receiving a first video stream from a video camera; means for receiving a second video stream from a depth camera, wherein a pixel mapping between the video camera and the depth camera is known, and wherein said video camera has an update rate greater than that of said depth camera; means for measuring optical flow in successive frames of said first video stream; means for extracting a portion of the optical flow attributable to depth change; means for calculating a scaling factor for each pixel in successive frames of said first video stream to determine whether a depth change has occurred; means for applying a perspective depth correction to each pixel having a depth change, said perspective depth correction being based upon the depth of the corresponding pixel in the most recent frame from the second video stream; and means for outputting a combined video stream having an update rate of the video camera and depth information from said depth camera. ### Claim 14 (depends on 13) The apparatus as claimed in claim 13, wherein images from said first video stream and images from said second video stream are aligned either optically or through a remapping process. ### Claim 15 (depends on 13) The apparatus as claimed in claim 13, wherein extracting a portion of the optical flow attributable to depth change comprises: applying Helmholtz-Hodge decomposition to decompose a two-dimensional optical flow into a divergence-free vector field and a divergence field; ignoring the divergence-free vector field; and testing the divergence field to determine whether divergence is caused by the depth change of a rigid object. ### Claim 16 (depends on 15) The apparatus as claimed in claim 15, wherein the testing is carried out by comparing unit structures formed by pixels and four spatial neighbors thereof in successive frames to determine whether the unit structures have changed in either size or orientation. ### Claim 17 (depends on 13) The apparatus as claimed in claim 13, further comprising: means for providing spatial and temporal neighboring pixels with the same depth correction when they have the same color. ### Claim 18 A computer program product comprising a non-transitory computer-readable storage medium bearing computer program code embodied therein for use with an apparatus, the computer program code comprising code for performing at least the following: receiving a first video stream from a video camera; receiving a second video stream from a depth camera, wherein a pixel mapping between the video camera and the depth camera is known, and wherein said video camera has an update rate greater than that of said depth camera; measuring optical flow in successive frames of said first video stream; extracting a portion of the optical flow attributable to depth change; calculating a scaling factor for each pixel in successive frames of said first video stream to determine whether a depth change has occurred; applying a perspective depth correction to each pixel having a depth change, said perspective depth correction being based upon the depth of the corresponding pixel in the most recent frame from the second video stream; and outputting a combined video stream having an update rate of the video camera and depth information from said depth camera. ### Claim 19 (depends on 18) The computer program product as claimed in claim 18, wherein images from said first video stream and images from said second video stream are aligned either optically or through a remapping process. ### Claim 20 (depends on 18) The computer program product as claimed in claim 18, wherein extracting a portion of the optical flow attributable to depth change comprises: applying Helmholtz-Hodge decomposition to decompose a two-dimensional optical flow into a divergence-free vector field and a divergence field; ignoring the divergence-free vector field; and testing the divergence field to determine whether divergence is caused by the depth change of a rigid object. ### Claim 21 (depends on 20) The computer program product as claimed in claim 20, wherein the testing is carried out by comparing unit structures formed by pixels and four spatial neighbors thereof in successive frames to determine whether the unit structures have changed in either size or orientation. ### Claim 22 (depends on 18) The computer program product as claimed in claim 18, further comprising code for performing: providing spatial and temporal neighboring pixels with the same depth correction when they have the same color.