Photorealistic synthesis of agents in traffic scenes

A system for synthesizing an image, comprising:

• a hardware processor;
• and
• a memory storing a computer program which, when executed by the hardware processor, causes the hardware processor to:
• extract agent neural radiance fields (NeRFs) from driving video logs;
• store agent NeRFs in a database;
• for a driving video log to be edited, extract a scene NeRF and agent NeRFs from the driving video log to be edited;
• select one or more agent NeRFs from the database to insert into or replace existing agents in a traffic scene of the driving video log based on photorealism criteria;
• edit the traffic scene by at least one of inserting a selected agent NeRF into the traffic scene, replacing existing agents in the traffic scene with the selected agent NeRF, or removing one or more existing agents from the traffic scene;
• and
• synthesize an image of the traffic scene by composing edited agent NeRFs with the scene NeRF and performing volume rendering.

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method for synthesizing an image, the method comprising:

• extract agent neural radiance fields (NeRFs) from driving video logs;
• store agent NeRFs in a database;
• for a driving video log to be edited, extract a scene NeRF and agent NeRFs from the driving video log to be edited;
• select one or more agent NeRFs from the database to insert into or replace existing agents in a traffic scene of the driving video log based on photorealism criteria;
• edit the traffic scene by at least one of inserting a selected agent NeRF into the traffic scene, replacing existing agents in the traffic scene with the selected agent NeRF, or removing one or more existing agents from the traffic scene;
• and
• synthesize an image of the traffic scene by composing edited agent NeRFs with the scene NeRF and performing volume rendering.