# US 2025/0148736 A1 · Photorealistic synthesis of agents in traffic scenes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148736 A1
- Document ID: US20250148736A1
- Status: Published application
- Kind: A1
- Application number: 18/924,258
- Filed: 2024-10-23
- Published: 2025-05-08
- Earliest claimed: 2023-11-07
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Bingbing Zhuang; Ziyu Jiang; Manmohan Chandraker; Shanlin Sun
- Web page: https://patentgrep.com/patent/US20250148736A1

## CPC

- G06T 15/08: Volume rendering
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/90: Determination of colour characteristics
- G06T 15/005: {General purpose rendering architectures}
- G06T 15/06: Ray-tracing
- G06T 15/20: Perspective computation
- G06T 15/205: {Image-based rendering}
- G06T 15/506: {Illumination models}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30236: Traffic on road, railway or crossing
- G06T 2207/30241: Trajectory
- G06T 2219/2008: Assembling, disassembling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A computer-implemented method for synthesizing an image includes extracting agent neural radiance fields (NeRFs) from driving video logs and storing agent NeRFs in a database. For a driving video log to be edited, a scene NeRF and agent NeRFs are extracted from the driving video log to be edited. One or more agent NeRFs are selected from the database to insert into or replace existing agents in a traffic scene of the driving video log based on photorealism criteria. The traffic scene is edited by inserting a selected agent NeRF into the traffic scene, replacing existing agents in the traffic scene with the selected agent NeRF, or removing one or more existing agents from the traffic scene. An image of the edited traffic scene is synthesized by composing edited agent NeRFs with the scene NeRF and performing volume rendering.

## Claims

### Claim 1

A computer-implemented method for synthesizing an image, comprising:
extracting agent neural radiance fields (NeRFs) from driving video logs;
storing agent NeRFs in a database;
for a driving video log to be edited, extracting a scene NeRF and agent NeRFs from the driving video log to be edited;
selecting one or more agent NeRFs from the database to insert into or replace existing agents in a traffic scene of the driving video log based on photorealism criteria;
editing the traffic scene by at least one of inserting a selected agent NeRF into the traffic scene, replacing existing agents in the traffic scene with a selected agent NeRF, or removing one or more existing agents from the traffic scene; and
synthesizing an image of an edited traffic scene by composing edited agent NeRFs with the scene NeRF and performing volume rendering.

### Claim 2 (depends on 1)

The method of claim 1, wherein the photorealism criteria include at least one of:
view consistency between inserted agents and the traffic scene, illumination consistency between inserted agents and the traffic scene, or shadow consistency between inserted agents and the traffic scene.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving outputs from an external behavior simulator indicating insertion locations, orientations, and driving trajectories for agents to be inserted into the traffic scene.

### Claim 4 (depends on 1)

The method of claim 1, wherein editing the traffic scene comprises:
randomly identifying one or more existing agents for removal from the traffic scene.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
representing each agent as a learnable latent code; and
applying a hyper-network to map the latent code to parameters of a feature grid for each agent.

### Claim 6 (depends on 5)

The method of claim 5, further comprising: using the feature grid to generate feature vectors for three-dimensional points associated with an agent.

### Claim 7 (depends on 6)

The method of claim 6, wherein synthesizing the image comprises:
decoding the feature vectors using multilayer perceptrons to determine density and color for points along rendering rays; and
performing volume rendering using the density and color.

### Claim 8

A system for synthesizing an image, comprising:
a hardware processor; and
a memory storing a computer program which, when executed by the hardware processor, causes the hardware processor to:
extract agent neural radiance fields (NeRFs) from driving video logs;
store agent NeRFs in a database;
for a driving video log to be edited, extract a scene NeRF and agent NeRFs from the driving video log to be edited;
select one or more agent NeRFs from the database to insert into or replace existing agents in a traffic scene of the driving video log based on photorealism criteria;
edit the traffic scene by at least one of inserting a selected agent NeRF into the traffic scene, replacing existing agents in the traffic scene with the selected agent NeRF, or removing one or more existing agents from the traffic scene; and
synthesize an image of the traffic scene by composing edited agent NeRFs with the scene NeRF and performing volume rendering.

### Claim 9 (depends on 8)

The system of claim 8, wherein the photorealism criteria include at least one of:
view consistency between inserted agents and the traffic scene, illumination consistency between inserted agents and the traffic scene, or shadow consistency between inserted agents and the traffic scene.

### Claim 10 (depends on 8)

The system of claim 8, wherein the computer program further causes the hardware processor to:
receive outputs from an external behavior simulator indicating insertion locations, orientations, and driving trajectories for agents to be inserted into the traffic scene.

### Claim 11 (depends on 8)

The system of claim 8, wherein editing the traffic scene comprises:
randomly identifying one or more existing agents for removal from the traffic scene.

### Claim 12 (depends on 8)

The system of claim 8, wherein the computer program further causes the hardware processor to:
represent each agent as a learnable latent code; and
apply a hyper-network to map the latent code to parameters of a feature grid for each agent.

### Claim 13 (depends on 12)

The system of claim 12, wherein the computer program further causes the hardware processor to:
use the feature grid to generate feature vectors for three-dimensional points associated with an agent.

### Claim 14 (depends on 13)

The system of claim 13, wherein synthesizing the image comprises:
decoding the feature vectors using multilayer perceptrons to determine density and color for points along rendering rays; and
performing volume rendering using the density and color.

### Claim 15

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method for synthesizing an image, the method comprising:
extract agent neural radiance fields (NeRFs) from driving video logs;
store agent NeRFs in a database;
for a driving video log to be edited, extract a scene NeRF and agent NeRFs from the driving video log to be edited;
select one or more agent NeRFs from the database to insert into or replace existing agents in a traffic scene of the driving video log based on photorealism criteria;
edit the traffic scene by at least one of inserting a selected agent NeRF into the traffic scene, replacing existing agents in the traffic scene with the selected agent NeRF, or removing one or more existing agents from the traffic scene; and
synthesize an image of the traffic scene by composing edited agent NeRFs with the scene NeRF and performing volume rendering.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the photorealism criteria include at least one of:
view consistency between inserted agents and the traffic scene, illumination consistency between inserted agents and the traffic scene, or shadow consistency between inserted agents and the traffic scene.

### Claim 17 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises:
receiving outputs from an external behavior simulator indicating insertion locations, orientations, and driving trajectories for agents to be inserted into the traffic scene.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein editing the traffic scene comprises:
randomly identifying one or more existing agents for removal from the traffic scene.

### Claim 19 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises:
representing each agent as a learnable latent code; and
applying a hyper-network to map the latent code to parameters of a feature grid for each agent.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium of claim 19, wherein synthesizing the image comprises:
using the feature grid to generate feature vectors for three-dimensional points associated with each agent;
decoding the feature vectors using multilayer perceptrons to determine density and color for points along rendering rays; and
performing volume rendering using the density and color.
