Deep novel view and lighting synthesis from sparse images

A non-transitory computer storage medium storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:

• generating a plurality of feature maps based on a plurality of images of an object, wherein each image of the plurality of images corresponds to one of a plurality of viewpoints and one of a plurality of lighting directions;
• appending the plurality of lighting directions to a sweeping plane volume that is generated based on a size of the object, a plurality of distances from the object to the plurality of viewpoints, and a number of depth planes in a plurality of depth planes;
• generating a cost volume by calculating a mean and a variance of a plurality of volumes that is generated by warping the plurality of feature maps;
• generating a relit image for each depth plane of the plurality of depth planes based on the sweeping plane volume and a novel lighting direction, wherein each relit image includes a corresponding plurality of pixels;
• calculating a depth probability for each pixel of the corresponding plurality of pixels in each relit image based on the cost volume;
• and
• generating a novel image of the object based on a weighted sum of the relit images and the calculated depth probabilities, the novel image corresponding to the novel viewpoint and the novel lighting direction.

A computerized system comprising:

• a feature map generating means for generating a plurality of feature maps based on a plurality of images of an object, wherein each image of the plurality of images corresponds to one of a plurality of viewpoints and one of a plurality of lighting directions;
• a warping means for warping the plurality of images to generate a sweeping plane volume, and for warping the plurality of feature maps to generate a cost volume, wherein the sweeping plane volume is generated based on a size of the object, a plurality of distances from the object to the plurality of viewpoints, and a number of depth planes in a plurality of depth planes;
• a relit image generating means for generating a relit image for each depth plane of the plurality of depth planes based on the sweeping plane volume and a novel lighting direction, wherein each relit image includes a corresponding plurality of pixels;
• a depth probability calculating means for calculating a depth probability for each pixel of the corresponding plurality of pixels in each relit image based on the cost volume;
• and
• a novel image generating means for generating the novel image of the object based on a weighted sum of the relit images and the calculated depth probabilities, the novel image corresponding to the novel viewpoint and the novel lighting direction.