# US 2021/0241519 A1 · Inverse Path Tracing for Material and Lighting Estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241519 A1 - Document ID: US20210241519A1 - Status: Published application - Kind: A1 - Application number: 17/178,122 - Filed: 2021-02-17 - Published: 2021-08-05 - Earliest claimed: 2019-03-05 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Anton S. Kaplanyan; Dejan Azinovic; Matthias Niessner; Tzu-Mao Li - Web page: https://patentgrep.com/patent/US20210241519A1 ## CPC - G06T 15/20: Perspective computation - G06T 15/506: {Illumination models} - G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In one embodiment, a computing system accesses a three-dimensional (3D) model of an environment, the 3D model comprising a virtual representation of an object in the environment. The computing system accesses an image of the object captured by a camera from a camera pose. The computing system accesses light source parameters associated with a virtual representation of a light source in the environment. The computing system renders, using the 3D model, pixels associated with the virtual representation of the object based on the light source parameters, the pixels being rendered from a virtual perspective corresponding to the camera pose. The computing system determines updated light source parameters based on a comparison of the rendered pixels to corresponding pixels located in the image of the object. ## Claims ### Claim at position 1 1-19. (canceled) ### Claim 20 A method comprising, by a computing system: accessing a three-dimensional (3D) model of an environment comprising at least a virtual representation of an object in the environment; accessing an image of the object captured by a camera from a camera pose; accessing one or more first light source parameters associated with a virtual representation of a light source in the environment; rendering, using the 3D model, one or more pixels associated with the virtual representation of the object from a virtual perspective corresponding to the camera pose based on the one or more first light source parameters; comparing the rendered one or more pixels to one or more corresponding pixels located in the image of the object to determine a difference between the one or more rendered pixels and the one or more corresponding pixels located in the image of the object; and determining one or more updated light source parameters associated with the virtual representation of the light source based on the difference. ### Claim 21 (depends on 20) The method of claim 20, further comprising: accessing a plurality of images of the object captured by the camera from a plurality of camera poses; rendering, using the 3D model, the one or more pixels associated with the virtual representation of the object based on the one or more first light source parameters associated with the virtual representation of the light source in the environment, the one or more pixels being rendered from a virtual perspective corresponding to each of the plurality of camera poses; and determining one or more updated light source parameters associated with the virtual representation of the light source in the environment based on a comparison of the rendered one or more pixels to each of one or more corresponding pixels located in the plurality of images of the object. ### Claim 22 (depends on 20) The method of claim 20, wherein accessing the one or more first light source parameters comprises one or more of performing an initial guess of the one or more first light source parameters or retrieving one or more stored values for the one or more first light source parameters. ### Claim 23 (depends on 20) The method of claim 20, further comprising: accessing one or more second light source parameters associated with the virtual representation of the light source, wherein the one or more second light source parameters correspond to the one or more updated light source parameters for the virtual representation of the light source; rendering, using the 3D model, one or more second pixels associated with the virtual representation of the object based on the one or more second light source parameters associated with the virtual representation of the light source, the one or more second pixels being rendered from the virtual perspective corresponding to the camera pose; and determining one or more third light source parameters associated with the light source based on a comparison of the rendered one or more second pixels to the one or more corresponding pixels located in the image of the object. ### Claim 24 (depends on 20) The method of claim 20, wherein the determination of the one or more updated light source parameters is responsive to determining the difference between the rendered one or more pixels and the one or more corresponding pixels exceeds a threshold difference. ### Claim 25 (depends on 20) The method of claim 20, wherein the determination of the updated light source parameters associated with the light source results in the updated light source parameters equal to the one or more first light source parameters responsive to determining the difference between the rendered one or more pixels and the one or more corresponding pixels is below a threshold difference. ### Claim 26 (depends on 20) The method of claim 20, further comprising generating a complete frame representative of the image of the object captured by the camera from the camera pose based on the rendered one or more pixels. ### Claim 27 (depends on 20) The method of claim 20, further comprising: accessing one or more first material parameters associated with a material property for the virtual representation of the object, wherein the rendering of the one or more pixels associated with the virtual representation of the object is further based on the one or more first material parameters; and determining one or more updated material parameters associated with the material property for the virtual representation of the object based on a comparison of the rendered one or more pixels to one or more corresponding pixels located in the image of the object. ### Claim 28 (depends on 27) The method of claim 27, wherein accessing the one or more first material parameters comprises one or more of performing an initial guess of the one or more first material parameters or retrieving one or more stored values for the one or more first material parameters. ### Claim 29 (depends on 27) The method of claim 27, wherein the determination of the one or more updated material parameters is responsive to determining a difference between the rendered one or more pixels and the one or more corresponding pixels exceeds a threshold difference. ### Claim 30 (depends on 27) The method of claim 27, wherein the determination of the one or more updated material parameters is based on the difference between the rendered one or more pixels and the one or more corresponding pixels. ### Claim 31 (depends on 27) The method of claim 27, wherein the determination of the updated material parameters associated with the material property for the virtual representation of the object results in the updated material parameters equal to the one or more first material parameters responsive to determining the difference between the rendered one or more pixels and the one or more corresponding pixels is below a threshold difference. ### Claim 32 (depends on 27) The method of claim 27, further comprising: accessing one or more second material parameters associated with the material property for the virtual representation of the object, wherein the one or more second material parameters correspond to the one or more updated material parameters for the virtual representation of the object; rendering, using the 3D model, one or more second pixels associated with the virtual representation of the object based on the one or more second material parameters associated with the virtual representation of the object, the one or more second pixels being rendered from the virtual perspective corresponding to the camera pose; and determining one or more third material parameters associated with the material property for the virtual representation of the object based on a comparison of the rendered one or more second pixels to the one or more corresponding pixels located in the image of the object. ### Claim 33 (depends on 27) The method of claim 27, wherein the determining the one or more updated material parameters is based on an optimization algorithm. ### Claim 34 One or more computer-readable non-transitory storage media embodying software that is operable when executed to: access a three-dimensional (3D) model of an environment comprising at least a virtual representation of an object in the environment; access an image of the object captured by a camera from a camera pose; access one or more first light source parameters associated with a virtual representation of a light source in the environment; render, using the 3D model, one or more pixels associated with the virtual representation of the object from a virtual perspective corresponding to the camera pose based on the one or more first light source parameters; compare the rendered one or more pixels to one or more corresponding pixels located in the image of the object to determine a difference between the one or more rendered pixels and the one or more corresponding pixels located in the image of the object; and determine one or more updated light source parameters associated with the virtual representation of the light source based on the difference. ### Claim 35 (depends on 34) The media of claim 34, wherein the software is further operable when executed to: access a plurality of images of the object captured by the camera from a plurality of camera poses; render, using the 3D model, the one or more pixels associated with the virtual representation of the object based on the one or more first light source parameters associated with the virtual representation of the light source in the environment, the one or more pixels being rendered from a virtual perspective corresponding to each of the plurality of camera poses; and determine one or more updated light source parameters associated with the virtual representation of the light source in the environment based on a comparison of the rendered one or more pixels to each of one or more corresponding pixels located in the plurality of images of the object. ### Claim 36 (depends on 34) The media of claim 34, wherein accessing the one or more first light source parameters comprises one or more of performing an initial guess of the one or more first light source parameters or retrieving one or more stored values for the one or more first light source parameters. ### Claim 37 A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to: access a three-dimensional (3D) model of an environment comprising at least a virtual representation of an object in the environment; access an image of the object captured by a camera from a camera pose; access one or more first light source parameters associated with a virtual representation of a light source in the environment; render, using the 3D model, one or more pixels associated with the virtual representation of the object from a virtual perspective corresponding to the camera pose based on the one or more first light source parameters; compare the rendered one or more pixels to one or more corresponding pixels located in the image of the object to determine a difference between the one or more rendered pixels and the one or more corresponding pixels located in the image of the object; and determine one or more updated light source parameters associated with the virtual representation of the light source based on the difference. ### Claim 38 (depends on 37) The system of claim 37, wherein the processors are further operable when executing the instructions to: receive one or more signals indicative of light intensities captured by one or more cameras, wherein the one or more signals are captured in a plurality of frames at a first frame rate; calculate light intensity metrics for each frame of the plurality of frames based on the one or more signals captured in the respective frames; detect one or more peaks based on the light intensity metrics associated with one or more frames of the plurality of frames, wherein the one or more frames were captured in a predetermined time period; determine a likelihood of perceptible flicker based on the detected one or more peaks; and generate a notification indicating the likelihood of perceptible flicker in response to a determination that the likelihood of perceptible flicker exceeds a predetermined threshold. ### Claim 39 (depends on 37) The system of claim 37, wherein accessing the one or more first light source parameters comprises one or more of performing an initial guess of the one or more first light source parameters or retrieving one or more stored values for the one or more first light source parameters.