Back to results
US 11,288,864 B2Granted
Methods and systems for producing content in multiple reality environments
AssigneeSimile Inc.
InventorsJames George, Alexander Porter, Timothy Scaffidi, Neil Purvey, Patricia Shiu
Life of this patent
Estimating the term…
Abstract
This disclosure contains methods and systems that allow filmmakers to port filmmaking and editing skills to produce content to be used in other environments, such as video game environments, and augmented reality, virtual reality, mixed reality, and non-linear storytelling environments.
Claims21
A method for producing a synthetic video image comprising:
- receiving one or more video and depth inputs from one or more respective camera systems capturing a scene one or more respective perspectives, wherein each video and depth input is captured by a respective camera system and includes a respective video stream of the scene captured from a respective perspective and a respective depth stream of the scene captured from the respective perspective;
- for each respective video and depth input, generating a depth and color stream corresponding to the respective perspective of the video and depth input based on the video stream and the depth stream, wherein each respective depth and color stream includes i) a color image stream including a sequence of color images derived from the video stream of the video and depth input and ii) a refined depth image stream corresponding to the color image stream that includes a sequence of dense refined depth images that are refined by reprojecting depth images from the depth stream into respective color images of the video stream, wherein each dense refined depth image includes a grid of depth pixels that each indicate a respective depth value and respective color values derived from a corresponding color image;
- generating a geometry video stream corresponding to the scene based on one or more depth and color streams respectively derived from the one or more color and depth input, wherein the geometry video stream includes a sequence of geometry frames, each geometry frame having embedded therein, a respective color image and a respective dense refined depth image from each of the one or more depth and color streams;
- selecting a surface reconstruction process to process the geometry video stream from a plurality of surface reconstruction processes based on a number of perspectives from which the scene is captured and whether a specific hardware capability is available to process the geometry video stream;
- generating a surface stream based on the geometry video stream in accordance with the selected surface reconstruction process, wherein the surface stream includes a geometry stream that defines a geometry of an object captured in the scene, and a texture stream that is time aligned with the geometry stream that defines a texture of a surface of the object;
- and
- outputting the surface stream to a buffer and/or a renderer.
4 more elements
The method of 8, wherein generating the voxel signed distance field includes:
- generating a voxel distance field based on each refined depth image corresponding to the frame, wherein the voxel distance field is a three-dimensional grid of voxels and each voxel indicates an average distance from the voxel to the implicit surface;
- signing at least a subset of voxels in the voxel distance field based on a respective classification of each voxel in the subset of voxels by:
- projecting a point representing the voxel into the geometry frame based on a global extrinsic calibration and a depth camera lens intrinsic calibration corresponding to the dense refined depth image;
- determining whether the point is behind a surface of the object depicted in the scene or in front of the surface;
- when the voxel is behind the surface, assigning a first sign to the average distance in the voxel;
- and
- when the voxel is in front of the surface, assigning a second sign that is opposite to the first sign to the average distance indicated in the voxel.
4 more elements
The method of 8, wherein determining a view-dependent texture includes:
- receiving the viewing position parameter from the renderer;
- projecting each fragment of the tessellated mesh into a color texture of the object derived from the color image streams corresponding to the tessellated mesh;
- and
- calculating a weighted average of each fragment based on the viewing position parameter, the perspective of the corresponding camera system, and a normal vector of the tessellated mesh.
1 more element
The method of 12, wherein generating the voxel signed distance field includes:
- generating a voxel distance field based on each refined depth image corresponding to the frame, wherein the voxel distance field is a three-dimensional grid of voxels and each voxel indicates an average distance from the voxel to the implicit surface;
- signing at least a subset of voxels in the voxel distance field based on a respective classification of each voxel in the subset of voxels by:
- projecting a point representing the voxel into the geometry frame based on a global extrinsic calibration and a depth camera lens intrinsic calibration corresponding to the dense refined depth image;
- determining whether the point is behind a surface of the object depicted in the scene or in front of the surface;
- when the voxel is behind the surface, assigning a first sign to the average distance in the voxel;
- and
- when the voxel is in front of the surface, assigning a second sign that is opposite to the first sign to the average distance indicated in the voxel.
4 more elements
Dependencies are inferred from the claim text.