Computer-generated image processing including volumetric scene reconstruction

A computer-implemented method of processing images from a main imaging device using capture device inputs from capture devices, the method comprising acts of:

• obtaining a main video sequence using a main camera capturing color images of objects visible in a live action scene including capturing a primary object in color visible in the live action scene, wherein the main video sequence comprises a main pixel array including pixel color values from the color images captured by the main camera at a main camera view of the live action scene;
• obtaining, concurrently with obtaining the main video sequence, additional image data of the live action scene using at least one ancillary device capable of capturing color from the live action scene, wherein the at least one ancillary device is positioned to capture visual information from an obscured object that is obscured, at least in part, from the main camera view by the primary object;
• mapping the additional image data into corresponding volumetric data according to main camera rays from the main camera and the additional image data captured by the ancillary devices;
• determining one or more portions of the main pixel array associated with the primary object to be replaced;
• digitally removing the primary object in a synthetic image by replacing the pixel color values of the primary object captured by the main camera, with replacement pixel color values associated with the obscured object, wherein replacement pixel color values of the synthetic pixel array comprise pixel color values of pixels of portions of the obscured object captured by the at least one ancillary device and pixel color values of pixels from the volumetric data;
• and
• merging computer-generated content with the synthetic image to generate an output image.