Methods and systems for producing content in multiple reality environments

A method for producing a synthetic video image comprising:

• receiving video and depth inputs a camera system capturing a scene from a perspective, wherein the video and depth input is captured by a respective camera system and includes a video stream of the scene captured from the perspective and a depth stream of the scene captured from the perspective;
• generating a depth and color stream based on the video stream and the depth stream, wherein the depth and color stream includes i) a color image stream including a sequence of color images derived from the video stream, and ii) a refined depth image stream corresponding to the color image stream that includes a sequence of dense refined depth images that are refined by reprojecting depth images from the depth stream into respective color images of the video stream, wherein each dense refined depth image includes a grid of depth pixels that each indicate a respective depth value and respective color values derived from a corresponding color image;
• generating a geometry video stream corresponding to the scene based on the depth and color stream, wherein the geometry video stream includes a sequence of geometry frames, each geometry frame having embedded therein, a respective color image and a respective dense refined depth image from the depth and color stream;
• generating a surface stream based on the geometry video stream in accordance with a selected surface reconstruction process, wherein the surface stream includes a geometry stream that defines a geometry of an object captured in the scene, and a texture stream that is time aligned with the geometry stream that defines a texture of a surface of the object;
• and
• outputting the surface stream to a buffer and/or a renderer.