Computing images of controllable dynamic scenes

An apparatus comprising:

• at least one processor;
• a memory storing instructions that, when executed by the at least one processor, perform a method for computing an image of a dynamic 3D scene comprising a 3D object, comprising:
• receiving a description of a deformation of the 3D object, the description comprising a cage of 3D elements and associated animation data from a physics engine or an articulated object model;
• for a pixel of the image, computing a ray from a virtual camera through the pixel into the cage animated according to the animation data and computing a plurality of samples on the ray, each sample being a 3D position and view direction in one of the 3D elements;
• computing a transformation of the samples into a canonical version of the cage to produce transformed samples, the canonical version of the cage representing the 3D object in a rest state or an origin state;
• for each transformed sample, querying a learnt radiance field parameterization of the 3D scene to obtain a color value and an opacity value;
• and
• applying a volume rendering method to the color and opacity values to produce a pixel value of the image.

A computer-implemented method of computing an image of a dynamic 3D scene comprising a 3D object, the method comprising:

• receive a description of a deformation of the 3D object;
• for a pixel of the image, computing a ray from a virtual camera through the pixel into the description and computing a plurality of samples on the ray, each sample being a 3D position and view direction in a 3D element;
• compute a transformation of the samples into a canonical space to produce transformed samples, the canonical space representing the 3D object in a rest state or an origin state;
• for each transformed sample, query a cache of associations between 3D points in the canonical space and obtain color and opacity values;
• and
• apply a volume rendering method to the color and opacity values to produce a pixel value of the image.