# US 2026/0100204 A1 · Systems and methods for automated movie generation and editing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0100204 A1
- Document ID: US20260100204A1
- Status: Published application
- Kind: A1
- Application number: 19/348,778
- Filed: 2025-10-02
- Published: 2026-04-09
- Earliest claimed: 2024-10-03
- Assignees: Meta Platforms, Inc.
- Applicants: Meta Platforms, Inc.
- Inventors: Zecheng He; Samaneh Azadi; Bowen Shi; Apoorv Vyas; Ann Lee; Ishan Satish Misra; Peizhao Zhang; Roshan Rajesh Sumbaly; Yaniv Nechemia Taigman; Peter Vajda; Yi-Chiao Wu; Andros Tjandra; Wei-Ning Hsu; Amit Zohar; Animesh Sinha; Yuval Kirstain; Shelly Sheynin; Adam Polyak; Matthew Le; Juefei Xu; Haoyu Ma; Tingbo Hou
- Web page: https://patentgrep.com/patent/US20260100204A1

## CPC

- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06V 10/82: using neural networks
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel
- H04N 21/43072: {of multiple content streams on the same device}
- H04N 21/8113: {comprising music, e.g. song in MP3 format}
- H04N 21/816: {involving special video data, e.g 3D video}
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method to generate synchronized audio for a video includes receiving the video including a sequence of frames and receiving a text input describing at least one of a scene, an event, or a mood to be reflected in an audio track. The method also includes generating a latent audio representation via an audio generation model conditioned jointly on video embeddings associated with the sequence of frames and text embeddings associated with the text input. The method also includes decoding the latent audio representation to produce an audio track temporally aligned with the video and semantically consistent with the text input.

## Claims

### Claim 1

A method to generate synchronized audio for a video, the method comprising:
receiving the video comprising a sequence of frames;
receiving a text input describing one or more of a scene, an event, or a mood to be reflected in an audio track;
generating a latent audio representation via an audio generation model conditioned jointly on video embeddings associated with the sequence of frames and text embeddings associated with the text input; and
decoding the latent audio representation to produce an audio track temporally aligned with the video and semantically consistent with the text input.

### Claim 2 (depends on 1)

The method of claim 1, wherein the audio track comprises one or more of instrumental music, ambient sound, or sound effects.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
encoding the sequence of frames into video embeddings using a vision encoder, and encoding the text input into text embeddings using a language encoder.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
concatenating the video embeddings and text embeddings into a multimodal embedding sequence, wherein the audio generation model is conditioned on the multimodal embedding sequence.

### Claim 5 (depends on 1)

The method of claim 1, wherein decoding the latent audio representation comprises applying a variational autoencoder trained to reconstruct audio signals from compressed latent representations.

### Claim 6 (depends on 1)

The method of claim 1, wherein the audio context comprises at least one of: audio infilling, audio extension, or audio replacement for one or more frames of the video.

### Claim 7 (depends on 1)

The method of claim 1, wherein the semantic consistency is based on the correspondence between the generated audio and the text input using a contrastive audio-video-text pre-training model.

### Claim 8

An apparatus to generate synchronized audio for a video, the apparatus comprising:
one or more processors; and
one or more memories coupled with the one or more processors and storing processor-executable code that, when executed by the one or more processors, is configured to cause the apparatus to:
receive the video comprising a sequence of frames;
receive a text input describing one or more of a scene, an event, or a mood to be reflected in an audio track;
generate a latent audio representation via an audio generation model conditioned jointly on video embeddings associated with the sequence of frames and text embeddings associated with the text input; and
decode the latent audio representation to produce an audio track temporally aligned with the video and semantically consistent with the text input.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the audio track comprises one or more of instrumental music, ambient sound, or sound effects.

### Claim 10 (depends on 8)

The apparatus of claim 8, wherein execution of the processor-executable code further causes the apparatus to encode the sequence of frames into video embeddings using a vision encoder, and encoding the text input into text embeddings using a language encoder.

### Claim 11 (depends on 8)

The apparatus of claim 8, wherein execution of the processor-executable code further causes the apparatus to concatenate the video embeddings and text embeddings into a multimodal embedding sequence, wherein the audio generation model is conditioned on the multimodal embedding sequence.

### Claim 12 (depends on 8)

The apparatus of claim 8, wherein execution of the processor-executable code that causes the apparatus to decode the latent audio representation further causes the apparatus to apply a variational autoencoder trained to reconstruct audio signals from compressed latent representations.

### Claim 13 (depends on 8)

The apparatus of claim 8, wherein the audio context comprises at least one of: audio infilling, audio extension, or audio replacement for one or more frames of the video.

### Claim 14 (depends on 1)

The apparatus of claim 1, wherein the semantic consistency is based on the correspondence between the generated audio and the text input using a contrastive audio-video-text pre-training model.

### Claim 15

A non-transitory computer-readable medium having program code recorded thereon to generate synchronized audio for a video, the program code executed by one or more processors and comprising:
program code to receive the video comprising a sequence of frames;
program code to receive a text input describing one or more of a scene, an event, or a mood to be reflected in an audio track;
program code to generate a latent audio representation via an audio generation model conditioned jointly on video embeddings associated with the sequence of frames and text embeddings associated with the text input; and
program code to decode the latent audio representation to produce an audio track temporally aligned with the video and semantically consistent with the text input.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the audio track comprises one or more of instrumental music, ambient sound, or sound effects.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the program code further comprises program code to encode the sequence of frames into video embeddings using a vision encoder, and encoding the text input into text embeddings using a language encoder.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the program code further comprises program code to concatenate the video embeddings and text embeddings into a multimodal embedding sequence, wherein the audio generation model is conditioned on the multimodal embedding sequence.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the program code to decode the latent audio representation further comprises program code to apply a variational autoencoder trained to reconstruct audio signals from compressed latent representations.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the audio context comprises at least one of: audio infilling, audio extension, or audio replacement for one or more frames of the video.
