Inverse Path Tracing for Material and Lighting Estimation

A method comprising, by a computing system:

• accessing a three-dimensional (3D) model of an environment comprising at least a virtual representation of an object in the environment;
• accessing an image of the object captured by a camera from a camera pose;
• accessing one or more first light source parameters associated with a virtual representation of a light source in the environment;
• rendering, using the 3D model, one or more pixels associated with the virtual representation of the object from a virtual perspective corresponding to the camera pose based on the one or more first light source parameters;
• comparing the rendered one or more pixels to one or more corresponding pixels located in the image of the object to determine a difference between the one or more rendered pixels and the one or more corresponding pixels located in the image of the object;
• and
• determining one or more updated light source parameters associated with the virtual representation of the light source based on the difference.

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

• access a three-dimensional (3D) model of an environment comprising at least a virtual representation of an object in the environment;
• access an image of the object captured by a camera from a camera pose;
• access one or more first light source parameters associated with a virtual representation of a light source in the environment;
• render, using the 3D model, one or more pixels associated with the virtual representation of the object from a virtual perspective corresponding to the camera pose based on the one or more first light source parameters;
• compare the rendered one or more pixels to one or more corresponding pixels located in the image of the object to determine a difference between the one or more rendered pixels and the one or more corresponding pixels located in the image of the object;
• and
• determine one or more updated light source parameters associated with the virtual representation of the light source based on the difference.

A system comprising:

• one or more processors;
• and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
• access a three-dimensional (3D) model of an environment comprising at least a virtual representation of an object in the environment;
• access an image of the object captured by a camera from a camera pose;
• access one or more first light source parameters associated with a virtual representation of a light source in the environment;
• render, using the 3D model, one or more pixels associated with the virtual representation of the object from a virtual perspective corresponding to the camera pose based on the one or more first light source parameters;
• compare the rendered one or more pixels to one or more corresponding pixels located in the image of the object to determine a difference between the one or more rendered pixels and the one or more corresponding pixels located in the image of the object;
• and
• determine one or more updated light source parameters associated with the virtual representation of the light source based on the difference.