Method and apparatus for graphics rendering using a neural processing unit

A neural processing unit (NPU) comprising:

• a processor;
• and
• a memory storing instructions thereon, the instructions when executed by the processor configure the NPU to:
• receive reference information indicating three-dimensional (3D) properties of a 3D scene;
• receive from a graphics processing unit (GPU):
• a plurality of two-dimensional (2D) images of the 3D scene each rendered by the GPU for a respective viewpoint of the 3D scene, and
• rendering information including intermediate rendering results produced during rendering of the plurality of 2D images;
• and
• train, based on the plurality of 2D images, the reference information, and the rendering information, a neural network model at the NPU to infer new images of the 3D scene for respective viewpoints of the 3D scene different from the respective viewpoint of each 2D image of the plurality of 2D images rendered by the GPU.

A system for training a neural processing unit (NPU) for graphics rendering, the system comprising a graphics processing unit (GPU) communicatively linked to a neural processing unit (NPU), wherein

• the GPU is configured to:
• receive a plurality of viewpoints for a three-dimensional (3D) scene;
• render a plurality of two-dimensional (2D) images of the 3D scene corresponding to the plurality of viewpoints;
• and
• transmit the plurality of 2D images and rendering information to the NPU, the rendering information including intermediate rendering results produced during the rendering of the plurality of 2D images;
• and
• the NPU is configured to:
• receive the plurality of 2D images of the 3D scene from the GPU;
• receive reference information indicating 3D properties of the 3D scene;
• receive the rendering information from the GPU;
• train, based on the plurality of 2D images, the reference information, and the rendering information a neural network model to infer new images of the 3D scene for respective viewpoints of the 3D scene different from the respective viewpoint of each 2D image of the plurality of 2D images rendered by the GPU.