Editing neural radiance fields with neural basis decomposition

A non-transitory computer readable storage medium including instructions stored thereon which, when executed by a processor, cause the processor to:

• generate a plurality of rays of the scene from a set of input images, wherein the rays include three-dimensional coordinates and viewing angles;
• determine a volume density of the scene by inputting the three-dimensional coordinates from the rays into a first neural network to generate a 3D geometric representation of the object;
• generate an appearance decomposition by inputting the three-dimensional coordinates and the viewing angles of the rays into a second neural network;
• produce an editable synthesized object within the scene by combining the 3D geometric representation of the object and the appearance decomposition using a volume rendering technique.

A system comprising:

• one or more processors;
• and
• one or more non-transitory computer storage media storing instructions that, when used by the one or more processors, cause the one or more processors to perform operations comprising:
• receiving a request to edit an appearance decomposition generated by a neural decomposition rendering system including a plurality of neural networks, wherein the neural decomposition rendering system generates the appearance decomposition representing an appearance of an object in a scene separately from a 3D geometric representation of the object;
• obtaining an edited appearance decomposition based on the request;
• and
• generating a modified appearance decomposition based on the edited appearance decomposition.