Techniques for intelligent video highlight summarization

A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform operations comprising:

• accessing, by the computing system, a video source comprising one or more video clips, each clip of the one or more video clips comprising a series of images, audio, and text elements associated with the series of images;
• generating, by the computing system, a first set of embeddings based at least in part on the video source, the first set of embeddings comprising an aligned image portion, an audio portion and a text portion;
• receiving, by the computing system, a query providing guidance for extracting clips from the video source, the guidance comprising a first-part guidance and a second-part guidance, the first-part guidance comprising preferences for events, and the second-part guidance comprising criteria, and the criteria comprising a duration budget;
• generating, by the computing system, a second set of embeddings based at least in part on the query;
• and
• performing, by a machine learning (ML) model of the computing system, a video analysis comprising:
• dividing the first set of embeddings into a plurality of sections;
• determining a similarity score between each section of the plurality of sections and the second set of embeddings;
• and
• generating a summary of the one or more video clips using the first set of embeddings and the second set of embeddings, the summary including a subset of the one or more video clips being selected based at least in part on the first-part guidance and the second-part guidance.

A computing system, comprising:

• one or more processors;
• and
• one or more non-transitory computer readable media storing computer-executable instructions that, when executed by the one or more processors of the computing system, cause the computing system to:
• access, by the computing system, a video source comprising one or more video clips, each clip of the one or more video clips comprising a series of images, audio, and text elements associated with the series of images;
• generate, by the computing system, a first set of embeddings based at least in part on the video source, the first set of embeddings comprising an aligned image portion, an audio portion and a text portion;
• receive, by the computing system, a query providing guidance for extracting clips from the video source, the guidance comprising a first-part guidance and a second-part guidance, the first-part guidance comprising preferences for events, and the second-part guidance comprising criteria, and the criteria comprising a duration budget;
• generate, by the computing system, a second set of embeddings based at least in part on the query;
• and
• perform, by a machine learning (ML) model of the computing system, a video analysis comprising:
• dividing the first set of embeddings into a plurality of sections;
• determining a similarity score between each section of the plurality of sections and the second set of embeddings;
• and
• generating a summary of the one or more video clips using the first set of embeddings and the second set of embeddings, the summary including a subset of the one or more video clips being selected based at least in part on the first-part guidance and the second-part guidance.