# US 2026/0094440 A1 · Video timeline assembly based on text instructions using machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0094440 A1
- Document ID: US20260094440A1
- Status: Published application
- Kind: A1
- Application number: 18/900,232
- Filed: 2024-09-27
- Published: 2026-04-02
- Earliest claimed: 2024-09-27
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Fabian David CABA HEILBRON; Jui-Hsien WANG; Bryan RUSSELL; Luis Alejandro PARDO GONZALEZ; Josef SIVIC
- Web page: https://patentgrep.com/patent/US20260094440A1

## CPC

- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments are trained to generate a timeline of digital media assets based on natural language instructions using a machine learning model. The method may include receiving an input including digital media assets, an input visual timeline, and text input, where the text input indicates a natural language instruction describing a modification to the input visual timeline using the digital media assets. The disclosed systems and methods further comprise generating a first set of tokens for the digital media assets, a second set of tokens for the input visual timeline, and a third set of tokens for the text input. The disclosed systems and methods further comprise processing, by a large language model, the first set of tokens, the second set of tokens, and the third set of tokens to generate an output set of tokens and generating a reconstructed visual timeline using the output set of tokens.

## Claims

### Claim 1

A method comprising:
receiving an input including digital media assets, an input visual timeline, and text input, the text input indicating a natural language instruction for a modification to the input visual timeline using the digital media assets;
generating a first set of tokens for the digital media assets, a second set of tokens for the input visual timeline, and a third set of tokens for the text input;
processing, by a large language model, the first set of tokens, the second set of tokens, and the third set of tokens to generate an output set of tokens; and
generating a reconstructed visual timeline using the output set of tokens.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first set of tokens includes a token pair for each digital media asset of the digital media assets, wherein the token pair includes a first set of identifier tokens and a set of visual tokens representing features of a corresponding digital media asset, wherein the second set of tokens for the input visual timeline includes a second set of identifier tokens, and wherein the third set of tokens for the text input includes text tokens.

### Claim 3 (depends on 2)

The method of claim 2, wherein the second set of tokens indicates an order of a subset of the digital media assets in the input visual timeline, and wherein each identifier token of the second set of identifier tokens includes a pointer to a token from the first set of tokens indicating a corresponding digital media asset.

### Claim 4 (depends on 2)

The method of claim 2, wherein each identifier token of the first set of identifier tokens and the second set of identifier tokens is one of a file name or a numerical value.

### Claim 5 (depends on 2)

The method of claim 2, wherein generating the reconstructed visual timeline using the output set of tokens further comprises:
mapping each token in the output set of tokens to a corresponding identifier token from the first set of identifier tokens; and
rendering the output set of tokens to generate a video sequence.

### Claim 6 (depends on 1)

The method of claim 1, wherein the natural language instruction for the modification to the input visual timeline references one or more digital media assets of the digital media assets using at least one of: an identifier value, positional information within the input visual timeline, and a description of content of the one or more digital media assets.

### Claim 7 (depends on 2)

The method of claim 2, further comprising:
generating each visual token of the set of visual tokens representing the features of the corresponding digital media asset by:
passing a digital media asset through a visual encoder to generate visual features of the digital media asset; and
mapping, by a projection layer, the visual features of the digital media asset into a visual token matching a dimensionality of the large language model.

### Claim 8 (depends on 1)

The method of claim 1, wherein the large language model is trained using a training dataset, and wherein each training assembly task of the training dataset is created by:
extracting a plurality of training digital media assets from a data source of video sequences;
generating, using a transformation function, a training visual timeline from the plurality of training digital media assets, wherein the transformation function uses a training assembly task of a plurality of assembly tasks and a cue for referencing a digital media asset in the training visual timeline, and wherein the training assembly task of the plurality of assembly tasks is associated with an instruction template; and
generating, using the instruction template, a training natural language instruction including the training assembly task and the cue.

### Claim 9

A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
receiving an input including digital media assets, an input visual timeline, and text input, the text input indicating a natural language instruction for a modification to the input visual timeline using the digital media assets;
generating a first set of tokens for the digital media assets, a second set of tokens for the input visual timeline, and a third set of tokens for the text input;
processing, by a large language model, the first set of tokens, the second set of tokens, and the third set of tokens to generate an output set of tokens; and
generating a reconstructed visual timeline using the output set of tokens.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the first set of tokens includes a token pair for each digital media asset of the digital media assets, wherein the token pair includes a first set of identifier tokens and a set of visual tokens representing features of a corresponding digital media asset, wherein the second set of tokens for the input visual timeline includes a second set of identifier tokens, and wherein the third set of tokens for the text input includes text tokens.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the second set of tokens indicates an order of a subset of the digital media assets in the input visual timeline, and wherein each identifier token of the second set of identifier tokens includes a pointer to a token from the first set of tokens indicating a corresponding digital media asset.

### Claim 12 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein each identifier token of the first set of identifier tokens and the second set of identifier tokens is one of a file name or a numerical value.

### Claim 13 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the executable instructions to generate the reconstructed visual timeline using the output set of tokens further comprise:
mapping each token in the output set of tokens to a corresponding identifier token from the first set of identifier tokens; and
rendering the output set of tokens to generate a video sequence.

### Claim 14 (depends on 9)

The non-transitory computer-readable medium of claim 9 wherein the natural language instruction for the modification to the input visual timeline references one or more digital media assets of the digital media assets using at least one of: an identifier value, positional information within the input visual timeline, and a description of content of the one or more digital media assets.

### Claim 15 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the executable instructions further comprise:
generating each visual token of the set of visual tokens representing the features of the corresponding digital media asset by:
passing a digital media asset through a visual encoder to generate visual features of the digital media asset; and
mapping, by a projection layer, the visual features of the digital media asset into a visual token matching a dimensionality of the large language model.

### Claim 16 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the large language model is trained using a training dataset, and wherein each training assembly task of the training dataset is created by:
extracting a plurality of training digital media assets from a data source of video sequences;
generating, using a transformation function, a training visual timeline from the plurality of training digital media assets, wherein the transformation function uses a training assembly task of a plurality of assembly tasks and a cue for referencing a digital media asset in the training visual timeline, and wherein the training assembly task of the plurality of assembly tasks is associated with an instruction template; and
generating, using the instruction template, a training natural language instruction including the training assembly task and the cue.

### Claim 17

A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
receiving an input including digital media assets, an input visual timeline, and text input, the text input indicating a natural language instruction for a modification to the input visual timeline using the digital media assets;
generating a first set of tokens for the digital media assets, a second set of tokens for the input visual timeline, and a third set of tokens for the text input;
processing, by a large language model, the first set of tokens, the second set of tokens, and the third set of tokens to generate an output set of tokens; and
generating a reconstructed visual timeline using the output set of tokens.

### Claim 18 (depends on 17)

The system of claim 17, wherein the first set of tokens includes a token pair for each digital media asset of the digital media assets, wherein the token pair includes a first set of identifier tokens and a set of visual tokens representing features of a corresponding digital media asset, wherein the second set of tokens for the input visual timeline includes a second set of identifier tokens, and wherein the third set of tokens for the text input includes text tokens.

### Claim 19 (depends on 18)

The system of claim 18, wherein the second set of tokens indicates an order of a subset of the digital media assets in the input visual timeline, and wherein each identifier token of the second set of identifier tokens includes a pointer to a token from the first set of tokens indicating a corresponding digital media asset.

### Claim 20 (depends on 18)

The system of claim 18, wherein the operations further comprise:
generating each visual token of the set of visual tokens representing the features of the corresponding digital media asset by:
passing a digital media asset through a visual encoder to generate visual features of the digital media asset; and
mapping, by a projection layer, the visual features of the digital media asset into a visual token matching a dimensionality of the large language model.
