# US 12,217,349 B2 · Differentiable inverse rendering based on radiative backpropagation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,349 B2 - Document ID: US12217349B2 - Status: Granted - Kind: B2 - Application number: 17/923,822 - Filed: 2020-05-07 - Published: 2023-06-29 - Granted: 2025-02-04 - Earliest claimed: 2020-05-07 - Assignees: Ecole Polytechnique Federale De Lausanne (Epfl) - Applicants: Ecole Polytechnique Federale De Lausanne (Epfl) - Inventors: Wenzel Jakob; Merlin Nimier-David - Web page: https://patentgrep.com/patent/US12217349B2 ## CPC - G06T 15/005: {General purpose rendering architectures} - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G06T 15/06: Ray-tracing - G06T 15/50: Lighting effects - G06T 15/506: {Illumination models} - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30168: Image quality inspection - G06T 2210/21: Collision detection, intersection - G06T 2210/61: Scene description - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A computer-implemented inverse rendering method comprising: computing an adjoint image by differentiating an objective function that evaluates the quality of a rendered image, image elements of the adjoint image encoding the sensitivity of spatially corresponding image elements of the rendered image with respect to the objective function; sampling the adjoint image at a respective sample position to determine a respective adjoint radiance value associated with the respective sample position; emitting the respective adjoint radiance value into a scene model characterised by scene parameters; determining an interaction location of a respective incident adjoint radiance value with a surface and/or volume of the scene model; determining a respective incident radiance value or an approximation thereof at the interaction location; and updating a scene parameter gradient. ## Claims ### Claim 1 A computer-implemented inverse rendering method comprising: computing an adjoint image by differentiating an objective function that evaluates the quality of a rendered image, image elements of the adjoint image encoding the sensitivity of spatially corresponding image elements of the rendered image with respect to the objective function; sampling the adjoint image at a respective sample position to determine a respective adjoint radiance value associated with the respective sample position; emitting the respective adjoint radiance value into a scene model characterised by scene parameters; determining an interaction location of a respective incident adjoint radiance value with a surface and/or volume of the scene model; determining a respective incident radiance value or an approximation thereof at the interaction location, the respective incident radiance value expressing the amount of incident radiance that travels from the surrounding environment towards the interaction location; and updating a scene parameter gradient by using at least the interaction location and the respective incident adjoint radiance value and the respective incident radiance value or its approximation at the interaction location, wherein the scene parameter gradient is updated without generating a transcript of the computation of the adjoint image. ### Claim 2 (depends on 1) The method according to claim 1, wherein the updated scene parameter gradient is computed by invoking an adjoint shader with the interaction location, the respective incident radiance value at the interaction location, and/or the respective incident adjoint radiance value, and by depositing an adjoint shader output onto the scene parameter gradient to obtain the updated scene parameter gradient. ### Claim 3 (depends on 1) The method according to claim 1, wherein the method further comprises initialising the scene parameter gradient prior to updating it. ### Claim 4 (depends on 1) The method according to claim 1, wherein the sample position is an integer or fractional sample position. ### Claim 5 (depends on 1) The method according to claim 1, wherein the respective adjoint radiance value is emitted in a computer simulation. ### Claim 6 (depends on 5) The method according to claim 5, wherein the respective adjoint radiance value is emitted from a camera in the computer simulation. ### Claim 7 (depends on 1) The method according to claim 1, wherein the parameter gradient comprises a set of scene parameter derivates. ### Claim 8 (depends on 1) The method according to claim 1, wherein the method further comprises rendering the scene model using an initial set of scene parameters to obtain the rendered image. ### Claim 9 (depends on 1) The method according to claim 1, wherein the method further comprises determining whether or not the rendered image satisfies a given quality target, and computing the adjoint image in the case where the quality target is not satisfied. ### Claim 10 (depends on 1) The method according to claim 1, wherein the interaction location of the respective adjoint radiance value is determined by using a ray tracing algorithm to determine an intersection location of a surface of the scene model with an adjoint radiance ray, and/or the interaction location of the respective adjoint radiance value is determined by using a free-flight sampling algorithm to determine an intersection location of a volume of the scene model with an adjoint radiance ray. ### Claim 11 (depends on 1) The method according to claim 1, wherein the method further comprises simulating reflection of the respective adjoint radiance value from the surface and/or volume at the interaction location to further propagate reflected adjoint radiance recursively to account for inter-reflection between different parts of the scene model. ### Claim 12 (depends on 1) The method according to claim 1, wherein the method further comprises determining whether or not one or more path termination criteria have been met for the emitted respective adjoint radiance value. ### Claim 13 (depends on 1) The method according to claim 1, wherein the respective incident radiance value at the interaction location is an approximation, such as a predefined constant. ### Claim 14 (depends on 1) The method according to claim 1, wherein the respective adjoint radiance value is a positive value when one or more scene parameter values should be modified so that the rendered image should become brighter in a spatial location corresponding to a spatial location of the respective sample position of the adjoint image, and wherein the respective adjoint radiance value is a negative value when one or more scene parameter values should be modified so that the rendered image should become darker in a spatial location corresponding to a spatial location of the respective sample position of the adjoint image, or vice versa. ### Claim 15 (depends on 1) The method according to claim 1, wherein the scene parameters comprise pixelised and/or voxelised representations of the scene model. ### Claim 16 (depends on 1) The method according to claim 1, wherein the scene parameters comprise one or more of the following items: object shape, object position, object material, light source parameter(s), camera pose, and surface parameter(s). ### Claim 17 (depends on 1) The method according to claim 1, wherein the respective adjoint radiance value is determined and emitted individually for a plurality sample positions of the adjoint image, and updating the scene parameter gradient by using the interaction locations of a plurality of incident adjoint radiance values. ### Claim 18 (depends on 1) The method according to claim 1, wherein the rendered image and the adjoint image have the same size, and wherein a respective image element of the rendered image has a spatially corresponding sample in the adjoint image. ### Claim 19 (depends on 1) The method according to claim 1, wherein the method further comprises evaluating the quality of a rendered image by using an objective function. ### Claim 20 (depends on 19) The method according to claim 19, wherein the quality of the rendered image is evaluated by comparing the rendered image to a reference image by using the objective function. ### Claim 21 (depends on 1) The method according to claim 1, wherein the method further comprises modifying at least some of the scene parameters by using the updated scene parameter gradient. ### Claim 22 (depends on 1) A non-transitory computer program product comprising instructions for implementing the steps of the method according to claim 1 when loaded and run on computing means of a computing apparatus. ### Claim 23 An apparatus for carrying out an inverse rendering of an image, the apparatus being configured to perform operations comprising: compute an adjoint image by differentiating an objective function that evaluates the quality of a rendered image, image elements of the adjoint image encoding the sensitivity of spatially corresponding image elements of the rendered image with respect to the objective function; sample the adjoint image at a respective sample position to determine a respective adjoint radiance value associated with the respective sample position; emit the respective adjoint radiance value into a scene model characterised by scene parameters; determine an interaction location of a respective incident adjoint radiance value with a surface and/or volume of the scene model; determine a respective incident radiance value or an approximation thereof at the interaction location, the respective incident radiance value expressing the amount of incident radiance that travels from the surrounding environment towards the interaction location; and update a scene parameter gradient without generating a transcript of the computation of the adjoint image by using at least the interaction location and the respective incident adjoint radiance value and the respective incident radiance value or its approximation at the interaction location.