Unified diffusion transformer model based on coupled video and action diffusion

An apparatus, comprising:

• one or more processors coupled to one or more memories and configured to:
• send, to a machine learning model, an input comprising a current image;
• and
• obtain, from the machine learning model, an output comprising one or more of:
• an action by a robot, or
• a future image,
• wherein:
• the machine learning model comprises a unified diffusion transformer model trained based on a joint diffusion process comprising (1) an action diffusion process based on a first training dataset comprising action data, and (2) a video diffusion process based on a second training dataset comprising video data, and
• the machine learning model is capable of performing one or more of:
• generating a first prediction based on forward dynamics,
• generating a second prediction based on inverse dynamics,
• generating a third prediction based on marginal action distribution, or
• generating a fourth prediction based on marginal image distribution.

A method, comprising:

• sending, to a machine learning model, an input comprising a current image;
• and
• obtaining, from the machine learning model, an output comprising one or more of:
• an action by a robot, or
• a future image,
• wherein:
• the machine learning model comprises a unified diffusion transformer model trained based on a joint diffusion process comprising (1) an action diffusion process based on a first training dataset comprising action data, and (2) a video diffusion process based on a second training dataset comprising video data, and
• the machine learning model is capable of performing one or more of:
• generating a first prediction based on forward dynamics,
• generating a second prediction based on inverse dynamics,
• generating a third prediction based on marginal action distribution, or
• generating a fourth prediction based on marginal image distribution.