Method and System for Programmatic Video Prediction Using Large Language Models

A system for video frame prediction, comprising:

• a processor;
• and
• a memory having instructions stored thereon that, when executed by the processor, cause the system to:
• receive a plurality of video frames representing an initial portion of a scene;
• generate, via a foundation model, a perception program configured to extract attributes from each of the plurality of received video frames;
• generate, via the foundation model, a dynamics program that models the temporal evolution of the extracted attributes based on physical principles;
• generate, via the foundation model, a rendering program that maps future attribute states to predicted video frames;
• execute the perception program to extract latent state representations from the initial video frames, wherein the latent state representations are derived on the basis of the extracted attributes associated with each of the plurality of video frames;
• apply the dynamics program to the latent state representations to generate predicted latent states corresponding to future time steps;
• and
• execute the rendering program to transform the predicted latent states into future video frames to perform the video prediction;
• wherein the perception program, the dynamics program, and the rendering program are each synthesized in a programmatic, human-interpretable format, and the latent state representations comprise physics-grounded variables.

A non-transitory computer-readable storage medium embodied thereon a program executable by a processor for performing a method for video frame prediction, the method comprising:

• receiving a plurality of video frames representing an initial portion of a scene;
• generating, via a foundation model, a perception program configured to extract attributes from each of the plurality of received video frames;
• generating, via the foundation model, a dynamics program that models the temporal evolution of the extracted attributes based on physical principles;
• generating, via the foundation model, a rendering program that maps future attribute states to predicted video frames;
• executing the perception program to extract latent state representations from the initial video frames, wherein the latent state representations are derived on the basis of the extracted attributes associated with each of the plurality of video frames;
• applying the dynamics program to the latent state representations to generate predicted latent states corresponding to future time steps;
• and
• executing the rendering program to transform the predicted latent states into future video frames to perform the video frame prediction;
• wherein the perception program, dynamics program, and rendering program are each synthesized in a programmatic, human-interpretable format, and the latent state representations comprise physics-grounded variables.