System and method of multi-view reconstruction with user-selectable novel views

A method of reconstructing of a photo-realistic rendering of an event with user-selectable novel views, the method comprising:

• (a) imaging the event with a plurality of cameras for producing a plurality of image frames, the event comprising a foreground and an environment;
• (b) modeling, by a CEM module, the environment from image data of the image frames;
• (c) segmenting, by an FES module, the foreground from the environment from image data of the image frames and constructing a 3D data representation;
• (d) configuring and rendering, by a configuration engine, a replay of the event;
• (e) receiving, by a viewing device, the replay for display to a viewer;
• and
• (f) selecting, by a user using a path selection module on the viewing device, a viewing path comprising at least one novel view image;
• wherein the photo-realistic rendering requires less than 10% discrepancy between output pixel raster values of the novel view image and the image frames imaged by the cameras, the discrepancy based at least on a change in position or orientation of the novel view image and the image frames imaged by the cameras, the discrepancy calculated using a pixel-wise RMS comparison in RGB color space between a plurality of sections of the novel view image and a corresponding plurality of sections of the image frames, wherein RMS=√{square root over (|Rm−Rg|2+|Gm−Gg|2+|Bm−Bg|2)}, wherein R, G, B represent RGB values, Xm denotes an image pixel value from the novel view image and Xg represents an image pixel value from the image frames imaged by the cameras.

A non-transitory computer readable storage medium including instructions stored thereon which, when executed by a processor, cause the processor to:

• image the event with a plurality of cameras for producing a plurality of image frames, the event comprising a foreground and an environment;
• model, by a CEM module, the environment from image data of the image frames;
• segment, by an FES module, the foreground from the environment from image data of the image frames and constructing a 3D data representation;
• configure and render, by a configuration engine, a replay of the event;
• receive, by a viewing device, the replay for display to a viewer;
• and
• select, by a user using a path selection module on the viewing device, a viewing path comprising at least one novel view image;
• wherein the photo-realistic rendering comprises requires less than 10% discrepancy between output pixel raster values of the novel view image and the image frames imaged by the cameras, the discrepancy based at least on a change in position or orientation of the novel view image and the image frames imaged by the cameras, the discrepancy calculated using a pixel-wise RMS comparison in RGB color space between a plurality of sections of the novel view image and a corresponding plurality of sections of the image frames, wherein RMS=√{square root over (|Rm−Rg|2+|Gm−Gg|2+|Bm−Bg|2)}, wherein R, G, B represent RGB values, Xm denotes an image pixel value from the novel view image and Xg represents an image pixel value from the image frames imaged by the cameras.