Back to results
US 9,369,689 B1Granted
Lidar stereo fusion live action 3D model video reconstruction for six degrees of freedom 360° volumetric virtual reality video
AssigneeHypeVR
InventorsAnthony Tran, Jason Juang
Life of this patent
Estimating the term…
Abstract
A system for capturing live-action three-dimensional video is disclosed. The system includes pairs of stereo cameras and a LIDAR for generating stereo images and three-dimensional LIDAR data from which three-dimensional data may be derived. A depth-from-stereo algorithm may be used to generate the three-dimensional camera data for the three-dimensional space from the stereo images and may be combined with the three-dimensional LIDAR data taking precedence over the three-dimensional camera data to thereby generate three-dimensional data corresponding to the three-dimensional space.
Claims24
A system for capturing live action three-dimensional video comprising:
- a first and a second camera, making up a stereo camera pair, the first and second stereo cameras each having known imaging parameters such that a depth-from-stereo algorithm may be applied to corresponding sets of images captured using the stereo camera pair to generate three-dimensional camera data for a three-dimensional space viewed by the stereo camera pair;
- a LIDAR, having a known position relative to the stereo camera pair, configured to produce three-dimensional LIDAR data;
- a time-stamp device for applying a first time-stamp to the three-dimensional camera data and a second time-stamp to the three-dimensional LIDAR data;
- and
- a controller configured to
- use the depth-from-stereo algorithm to generate the three-dimensional camera data for the three-dimensional space;
- combine the three-dimensional camera data and the three-dimensional LIDAR data, with the three-dimensional LIDAR data taking precedence over the three-dimensional camera data to thereby generate three-dimensional data corresponding to the three-dimensional space;
- detect image seams between each of the sets of images using one of color matching and gradient detection;
- generate a low resolution version of each of the sets of images;
- select preliminary blend colors for each pixel on the image seams of the low resolution version of each of the sets of images;
- provide the preliminary blend colors as a baseline for use in selecting blend colors for use in performing color blending for the sets of images at captured resolution to thereby lower a search space for appropriate blend colors for the image seams between the sets of images at the captured resolution;
- select a blend color for each pixel along the image seams using the preliminary blend color that corresponds to the associated pixel at the captured resolution as a starting point for a search for an appropriate blend color for each pixel of the sets of images at the captured resolution;
- and
- perform color blending on the image seams using the selected blend color for each pixel.
11 more elements
A system for capturing live action three-dimensional video comprising:
- a plurality of stereo camera pairs, each made up of a first stereo camera and a second stereo camera each camera having known imaging parameters such that a depth-from-stereo algorithm may be applied to corresponding sets of images captured using each of the plurality of stereo camera pairs to generate three-dimensional camera data for a three-dimensional space viewed by the plurality of stereo camera pairs;
- a LIDAR, having a known position relative to the stereo camera pair, configured to produce three-dimensional LIDAR data;
- a time-stamp device for applying a first time-stamp to the three-dimensional camera data and a second time-stamp to the three-dimensional LIDAR data;
- and
- a controller configured to
- use the depth-from-stereo algorithm to generate the three-dimensional camera data for the three-dimensional space;
- combine the three-dimensional camera data and the three-dimensional LIDAR data, with the three-dimensional LIDAR data taking precedence over the three-dimensional camera data to thereby generate three-dimensional data corresponding to the three-dimensional space;
- detect image seams between each of the sets of images using one of color matching and gradient detection;
- generate a low resolution version of each of the sets of images;
- select preliminary blend colors for each pixel on the image seams of the low resolution version of each of the sets of images;
- provide the preliminary blend colors as a baseline for use in selecting blend colors for use in performing color blending for the sets of images at captured resolution to thereby lower a search space for appropriate blend colors for the image seams between the sets of images at the captured resolution;
- select a blend color for each pixel along the image seams using the preliminary blend color that corresponds to the associated pixel at the captured resolution as a starting point for a search for an appropriate blend color for each pixel of the sets of images at the captured resolution;
- and
- perform color blending on the image seams using the selected blend color for each pixel.
11 more elements
Apparatus comprising a non-transitory storage medium storing a program having instructions which when executed by a processor will cause the processor to generate three-dimensional data, the instructions of the program for:
- generating three-dimensional camera data for a three-dimensional space viewed by a plurality of stereo camera pairs, each made up of a first stereo camera and a second stereo camera each camera having known imaging parameters such that a depth-from-stereo algorithm may be applied to corresponding sets of images;
- producing three-dimensional LIDAR data using a LIDAR;
- applying a first time-stamp to the three-dimensional camera data and a second time-stamp to the three-dimensional LIDAR data;
- using the depth-from-stereo algorithm to generate the three-dimensional camera data for the three-dimensional space;
- combining the three-dimensional camera data and the three-dimensional LIDAR data, with the three-dimensional LIDAR data taking precedence over the three-dimensional camera data to thereby generate three-dimensional data corresponding to the three-dimensional space;
- detecting image seams between each of the sets of images using one of color matching and gradient detection;
- generating a low resolution version of each of the sets of images;
- selecting preliminary blend colors for each pixel on the image seams of the low resolution version of each of the sets of images;
- providing the preliminary blend colors as a baseline for use in selecting blend colors for use in performing color blending for the sets of images at captured resolution to thereby lower a search space for appropriate blend colors for the image seams between the sets of images at the captured resolution;
- selecting a blend color for each pixel along the image seams using the preliminary blend color that corresponds to the associated pixel at the captured resolution as a starting point for a search for an appropriate blend color for each pixel of the sets of images at the captured resolution;
- and
- performing color blending on the image seams using the selected blend color for each pixel.
9 more elements
Dependencies are inferred from the claim text.