# US 2026/0099987 A1 · 4d generative models from in the wild videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0099987 A1
- Document ID: US20260099987A1
- Status: Published application
- Kind: A1
- Application number: 19/352,319
- Filed: 2025-10-07
- Published: 2026-04-09
- Earliest claimed: 2024-10-09
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Koki NAGANO; Kaiwen JIANG; Shalini DE MELLO; Xueting LI
- Web page: https://patentgrep.com/patent/US20260099987A1

## CPC

- G06T 15/08: Volume rendering
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06T 13/205: {driven by audio data}
- H04N 21/816: {involving special video data, e.g 3D video}
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

At least one embodiment for generating 4D generative models from in the wild videos includes receiving a first 2D image frame, processing the first 2D image frame to generate 3D Gaussians, defining a set of motion basis features from the 3D Gaussians, receiving a second 2D image frame, generating a plurality of augmented images based on the first 2D image frame and the second 2D image frame, processing the plurality of augmented images to generate a plurality of motion features, constructing deformed 3D Gaussians from the motion basis features and the motion features, generating a rendered 2D image from the deformed 3D Gaussians, and generating a 4D representation using a neural network trained based on the rendered 2D image and the second 2D image frame.

## Claims

### Claim 1

A computer-implemented method for generating four-dimensional (4D) representation, the method comprising:
receiving a first two-dimensional (2D) image frame;
processing the first 2D image frame to generate a plurality of 3D Gaussians;
defining a set of motion basis features from the plurality of 3D Gaussians;
receiving a second 2D image frame;
generating a plurality of augmented images based on the first 2D image frame and the second 2D image frame;
processing the first 2D image frame, the second 2D image frame, and the plurality of augmented images to generate a plurality of motion features;
constructing a plurality of deformed 3D Gaussians from the motion basis features and the motion features;
generating a rendered 2D image from the plurality of deformed 3D Gaussians; and
generating a 4D representation using a trained neural network.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising transforming the first 2D image frame into a 3D representation by aligning features of the first 2D image frame along a plurality of orthogonal planes.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising generating a plurality of feature vectors by sampling a plurality of 3D points along rays and projecting each of the plurality of 3D points onto the plurality of orthogonal planes.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein constructing the plurality of deformed 3D Gaussians comprises translating the plurality of 3D Gaussians based on the plurality of motion features.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the trained neural network is trained by minimizing rendering loss between the rendered 2D image and the second 2D image frame.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein generating the motion features comprises processing the first 2D image frame, the second 2D image frame, and the plurality of augmented images using a vision transformer.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein generating the rendered 2D image comprises performing splatting using the plurality of deformed 3D Gaussians.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising generating a 4D video using the 4D representation.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein generating the 4D video comprises:
receiving an audio input;
concatenating the audio input and noise removing noise from the concatenated audio input and noise to generate denoised audio features; and
generating the 4D video based on the denoised audio features and the 4D representation.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein generating the 4D video comprises processing the 4D representation with a diffusion model.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein the diffusion model is trained by:
receiving an audio input;
generating noisy features by generating a sequence where, at each step, noise is added to the audio input; and
generating predicted denoised features by generating a sequence, where, at each step noise is iteratively removed.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10, wherein the diffusion model is trained by diffusion forcing.

### Claim 13 (depends on 10)

The computer-implemented method of claim 10, wherein the noise comprises Gaussian noise.

### Claim 14

One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the steps of:
receiving a first 2D image frame;
processing the first 2D image frame to generate a plurality of 3D Gaussians;
defining a set of motion basis features from the plurality of 3D Gaussians;
receiving a second 2D image frame;
generating a plurality of augmented images based on the first 2D image frame and the second 2D image frame;
processing the first 2D image frame, the second 2D image frame, and the plurality of augmented images to generate a plurality of motion features;
constructing a plurality of deformed 3D Gaussians from the motion basis features and the motion features;
generating a rendered 2D image from the plurality of deformed 3D Gaussians;
generating a 4D representation using a trained neural network.

### Claim 15 (depends on 14)

The one or more non-transitory computer-readable media of claim 14, wherein the steps further comprise transforming the first 2D image frame into a 3D representation by aligning the features of the first 2D image frame along a plurality of orthogonal planes.

### Claim 16 (depends on 15)

The one or more non-transitory computer-readable media of claim 15, wherein the steps further comprise generating a plurality of feature vectors by sampling a plurality of 3D points along rays and projecting each of the plurality of 3D points onto the plurality of orthogonal planes.

### Claim 17 (depends on 14)

The one or more non-transitory computer-readable media of claim 14, wherein constructing the plurality of deformed 3D Gaussians comprises translating the plurality of 3D Gaussians based on the plurality of motion features.

### Claim 18 (depends on 14)

The one or more non-transitory computer-readable media of claim 14, wherein training the trained neural network comprises minimizing rendering loss between the rendered 2D image and the second 2D image frame.

### Claim 19 (depends on 14)

The one or more non-transitory computer-readable media of claim 14, further comprising generating a 4D video using the 4D representation.

### Claim 20

A system, comprising:
one or more memories storing instructions; and
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform steps comprising:
receiving a first 2D image frame;
processing the first 2D image frame to generate a plurality of 3D Gaussians;
defining a set of motion basis features from the plurality of 3D Gaussians;
receiving a second 2D image frame;
generating a plurality of augmented images based on the first 2D image frame and the second 2D image frame;
processing the first 2D image frame, the second 2D image frame, and the plurality of augmented images to generate a plurality of motion features;
constructing a plurality of deformed 3D Gaussians from the motion basis features and the motion features;
generating a rendered 2D image from the plurality of deformed 3D Gaussians;
generating a 4D representation using a trained neural network.
