US 8,134,637 B2 · Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing

Abstract

An imaging system substantially simultaneously acquires z-depth and brightness data from first sensors, and acquires higher resolution RGB data from second sensors, and fuses data from the first and second sensors to model an RGBZ image whose resolution can be as high as resolution of the second sensors. Time correlation of captured data from first and second sensors is associated with captured image data, which permits arbitrary mapping between the two data sources, ranging from 1:many to many:1. Preferably pixels from each set of sensors that image the same target point are mapped. Many z-depth sensor settings may be used to create a static environmental model. Non-correlative and correlative filtering is carried out, and up-sampling to increase z-resolution occurs, from which a three-dimensional model is constructed using registration and calibration data.

Claims20

Dependent claims

For use with a sensor system that includes first sensors to acquire from a scene z-depth values and brightness data values at a first resolution, each of said first sensors occupying a first sensor area, and second sensors to acquire from said scene RGB data values at a higher second resolution, each of said second sensors occupying a second sensor area, said first and second sensors acquiring data values substantially simultaneously, a method to combine data values acquired by said first sensors with data values acquired by said second sensors such that an RGBZ image may be rendered whose resolution is higher than said first resolution, the method including the following steps:

(a) dynamically correlating data values acquired by said first sensors and data values acquired by said second sensors such that depth data computed for a higher resolution pixel are affected by data values from said second sensors;
and
(b) up-sampling and increasing resolution of data values acquired by said first sensors such that an RGBZ model may be rendered from data values so up-sampled and increased in resolution, resolution of said RGBZ model exceeding said first resolution of said first sensors.

A time-of-flight system that acquires z-depth values, brightness values, and RGB information values and outputs data values from which an RGBZ image may be rendered, the time-of-flight system including:

a plurality of first sensors to acquire from a scene z-depth values and brightness data values at a first resolution, each of said first sensors occupying a first sensor area;
a plurality of second sensors to acquire from said scene RGB data values at a higher second resolution, each of said second sensors occupying a second sensor area, said first and second sensors acquiring data values substantially simultaneously, said first sensors having a characteristic selected from a group consisting of (i) said first sensors are fabricated on a common integrated circuit array as said second sensors, (ii) said first sensors are fabricated on an integrated circuit array discrete from an integrated circuit array that includes said second sensors, and (iii) said first sensor area is greater than said second sensor area, and further including:
means for dynamically correlating data values acquired by said first sensors with data values acquired by said second sensors such that said RGBZ image may be rendered whose resolution is higher than said first resolution, in which RGB image data values from said second sensors affect depth data of a higher resolution pixel that is computed based on data values of said first sensor.

Dependencies are inferred from the claim text.

Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing

Life of this patent

Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing

Life of this patent

Abstract

Claims20