Machine learning-based summarizations and vector representations for identifying relevant video segments

Abstract

Approaches presented herein provide for the identification of relevant media content in response to a received prompt or query. A plurality of video files, or other instances of content, can be broken into segments that can each be analyzed by a vision language model (or other such mechanism) to generate text-based segment summaries with timestamps. A language model can then generate an overall summary for a video file based in part on the segment summaries and timestamps, and a vector representation may be generated that may also include image features or other aspects of the video file. The vector representation can be stored to a vector database. When a prompt or query is received that includes text, image, and/or video content, for example, a search vector can be generated that can be used to identify relevant results from the vector database. Relevant portions of the identified video files can then be provided for playback based in part upon the timestamps associated with those relevant portions.

Dependencies are inferred from the claim text.