4d generative models from in the wild videos

One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the steps of:

• receiving a first 2D image frame;
• processing the first 2D image frame to generate a plurality of 3D Gaussians;
• defining a set of motion basis features from the plurality of 3D Gaussians;
• receiving a second 2D image frame;
• generating a plurality of augmented images based on the first 2D image frame and the second 2D image frame;
• processing the first 2D image frame, the second 2D image frame, and the plurality of augmented images to generate a plurality of motion features;
• constructing a plurality of deformed 3D Gaussians from the motion basis features and the motion features;
• generating a rendered 2D image from the plurality of deformed 3D Gaussians;
• generating a 4D representation using a trained neural network.

A system, comprising:

• one or more memories storing instructions;
• and
• one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform steps comprising:
• receiving a first 2D image frame;
• processing the first 2D image frame to generate a plurality of 3D Gaussians;
• defining a set of motion basis features from the plurality of 3D Gaussians;
• receiving a second 2D image frame;
• generating a plurality of augmented images based on the first 2D image frame and the second 2D image frame;
• processing the first 2D image frame, the second 2D image frame, and the plurality of augmented images to generate a plurality of motion features;
• constructing a plurality of deformed 3D Gaussians from the motion basis features and the motion features;
• generating a rendered 2D image from the plurality of deformed 3D Gaussians;
• generating a 4D representation using a trained neural network.