Systems and methods for physically-based neural face shader via volumetric lightmaps

A method for rendering an image of a subject or an object, the method comprising:

• receiving, at a processor, one or more first input images including the subject or the object, a desired camera viewing angle of the subject, and a desired lighting condition;
• learning, via the processor, a set of reflectance representations using a first trained neural network, the set of skin reflectance representations based on the one or more first input images including the subject or the object and the desired camera viewing angle;
• generating, via the processor, a volumetric light map based on a set of indirect light representations learnt using a second trained neural network based on one or more of the one or more first input images, the desired camera viewing angle, and the desired lighting condition;
• generating, via the processor, a volumetric radiance field based on the set of skin reflectance representations, the volumetric light map, and the one or more first input images;
• and
• rendering, via a display communicatively coupled to the processor, a second image of the subject or the object at the desired camera viewing angle and the desired lighting condition.

A system, comprising:

• one or more memories storing instructions and storing a first trained neural network model and a second trained neural network model;
• one or more processors that are coupled to the one or more memories and when executing the instructions, are configured to:
• generate a set of reflectance representations corresponding to a subject in a first image, according to a first trained neural network model;
• generate a first volumetric light map including a set of indirect light representations corresponding to the subject and a first lighting condition, according to a second trained neural network model;
• and
• render one or more second images of the subject under a desired lighting condition based on the set of reflectance representations and the first volumetric light map.