Volumetric performance capture with neural rendering

A system comprising:

• a camera system having a plurality of infrared cameras;
• a light stage having a plurality of lights;
• and
• a computing device configured to:
• obtain, using the camera system and the light stage having the plurality of lights, a plurality of images that depict a subject from a plurality of viewpoints and under a plurality of lighting conditions;
• obtain, using the plurality of infrared cameras, depth data corresponding to the subject;
• based on the depth data corresponding to the subject, extract, using a neural network, a plurality of features of the subject from the plurality of images;
• map, using the neural network, the plurality of features of the subject into a texture space;
• and
• generate, using a neural renderer, an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view, wherein the neural renderer is configured to resample the features of the subject from the texture space to an image space to generate the output image.

A non-transitory computer-readable medium configured to store instructions, that when executed by a computing system comprising one or more processors, causes the computing system to perform operations comprising:

• obtaining, using a camera system and a light stage having a plurality of lights, a plurality of images that depict a subject from a plurality of viewpoints and under a plurality of lighting conditions;
• obtaining, using a plurality of infrared cameras, depth data corresponding to the subject;
• based on the depth data corresponding to the subject, extracting, using a neural network, a plurality of features of the subject from the plurality of images;
• mapping, using the neural network, the plurality of features of the subject into a texture space;
• and
• generating, using a neural renderer, an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view, wherein the neural renderer is configured to resample the features of the subject from the texture space to an image space to generate the output image.