US 12,718,565 B1 · System for generation of recap of long video

Abstract

Input data comprising video and associated audio is processed to generate recap data, such as video with audio that summarizes previous occurrence and entices a viewer. The input data is processed to generate dense text data that includes timestamps, dialogue, descriptions of scenes of the video, and so forth. The input data is also processed to generate summary data, such as scene, episode, and season summaries. A set of one or more machine learning (ML) systems use this as input to generate a narration script. Narration audio is generated from the narration script. The narration script is then used to generate sections of a recap such as an intro, teaser, detail, and outro. Each portion may include narration audio, clips of video, video clips of relevant dialog, background music, audio effects, and so forth. The duration of elements may be constrained based on the presentation of associated narration audio.

Claims20

Dependent claims

A system comprising:

one or more memories, storing computer-executable instructions;
and
one or more hardware processors to execute the computer-executable instructions to:
determine input data comprising video data with associated audio data;
determine dense text data (DTD) that is associated with the input data, wherein the DTD comprises text data that is indicative of one or more of:
dialog, speaker annotation indicative of who uttered dialog, people, action, event, or setting;
determine summary data that is associated with the input data, wherein the summary data summarizes a portion of the input data;
determine a narration script using a first set of prompts, the summary data, and at least a portion of the DTD as input to a first set of machine learning (ML) models;
determine segment data using a second set of prompts and the narration script as input to the first set of ML models;
and
determine recap data using a third set of prompts, the segment data, and the input data as input to the first set of ML models, wherein the recap data comprises audio and video.

8 more elements

A computer-implemented method comprising:

determining input data comprising video data with associated audio data;
determining dense text data (DTD) that is associated with the input data, wherein the DTD comprises text data that is indicative of one or more of:
dialog, speaker annotation indicative of who uttered dialog, people, action, event, or setting;
determining summary data that is associated with the input data, wherein the summary data summarizes a portion of the input data;
determining a narration script using a first set of prompts, the summary data, and at least a portion of the DTD as input to a first set of machine learning (ML) models;
determining segment data using a second set of prompts and the narration script as input to the first set of ML models;
and
determining recap data using a third set of prompts, the segment data, and the input data as input to the first set of ML models, wherein the recap data comprises audio and video.

5 more elements

Dependencies are inferred from the claim text.

System for generation of recap of long video

Life of this patent

System for generation of recap of long video

Life of this patent

Abstract

Claims20