Systems and methods for automated movie generation and editing

An apparatus to generate synchronized audio for a video, the apparatus comprising:

• one or more processors;
• and
• one or more memories coupled with the one or more processors and storing processor-executable code that, when executed by the one or more processors, is configured to cause the apparatus to:
• receive the video comprising a sequence of frames;
• receive a text input describing one or more of a scene, an event, or a mood to be reflected in an audio track;
• generate a latent audio representation via an audio generation model conditioned jointly on video embeddings associated with the sequence of frames and text embeddings associated with the text input;
• and
• decode the latent audio representation to produce an audio track temporally aligned with the video and semantically consistent with the text input.

A non-transitory computer-readable medium having program code recorded thereon to generate synchronized audio for a video, the program code executed by one or more processors and comprising:

• program code to receive the video comprising a sequence of frames;
• program code to receive a text input describing one or more of a scene, an event, or a mood to be reflected in an audio track;
• program code to generate a latent audio representation via an audio generation model conditioned jointly on video embeddings associated with the sequence of frames and text embeddings associated with the text input;
• and
• program code to decode the latent audio representation to produce an audio track temporally aligned with the video and semantically consistent with the text input.