Deformable neural radiance fields

A computer program product comprising a nontransitive storage medium, the computer program product including code that, when executed by processing circuitry of a computing device, causes the processing circuitry to perform a method, the method comprising:

• acquiring image data representing a plurality of images, each of the plurality of images including an image of a scene within an observation frame, the scene including a non-rigidly deforming object viewed from a respective perspective;
• generating a deformation model based on the image data, the deformation model describing movements made by the non-rigidly deforming object while the image data was generated, the deformation model being represented by a differentiable non-linear mapping between a position in the observation frame to a position in a canonical frame;
• and
• generating a neural radiance field based on positions and viewing directions of casted rays through the positions in the canonical frame, the neural radiance field providing a mapping between the positions and viewing directions to a color and optical density at each position in the observation frame, the color and optical density at each position in the observation frame enabling a viewing of the non-rigidly deforming object from a new perspective.

An electronic apparatus, the electronic apparatus comprising:

• memory;
• and
• controlling circuitry coupled to the memory, the controlling circuitry being configured to:
• acquire image data representing a plurality of images, each of the plurality of images including an image of a scene within an observation frame, the scene including a non-rigidly deforming object viewed from a respective perspective;
• generate a deformation model based on the image data, the deformation model describing movements made by the non-rigidly deforming object while the image data was generated, the deformation model being represented by a differentiable non-linear mapping between a position in the observation frame to a position in a canonical frame;
• and
• generate a neural radiance field based on positions and viewing directions of casted rays through the positions in the canonical frame, the neural radiance field providing a mapping between the positions and viewing directions to a color and optical density at each position in the observation frame, the color and optical density at each position in the observation frame enabling a viewing of the non-rigidly deforming object from a new perspective.