Deep video understanding with large language models

A system comprising:

• one or more memories;
• at least one processor each coupled to at least one of the memories and configured to perform operations comprising:
• for each frame of a plurality of frames of a content item:
• determining, based on (i) respective element types and respective locations for a respective first visual element and a respective second visual element within the frame and (ii) at least one of spatial rules, temporal rules, or contextual rules applied to the respective element types and the respective locations for the respective first visual element and the respective second visual element within the frame, a relationship between the respective first visual element and the respective second visual element,
• generating a respective visual prompt comprising a textual description of the relationship between the respective first visual element and the respective second visual element within the frame, and
• generating, based on at least one of an audio-to-text conversion of audio content associated with the frame or classification of aural elements of the audio content, a respective audio prompt comprising a textual description of the audio content associated with the frame;
• and
• receiving, based on (i) the respective visual prompt and the respective audio prompt for each frame of the plurality of frames and (ii) an additional prompt for each frame generated from at least one of metadata associated with the content item, closed captioning data associated with the content item, or audio descriptive data associated with the content item input to a large language model (LLM) trained to output descriptive information for content items, a description of the content item.

A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:

• for each frame of a plurality of frames of a content item:
• determining based on (i) respective element types and respective locations for a respective first visual element and a respective second visual element within the frame and (ii) at least one of spatial rules, temporal rules, or contextual rules applied to the respective element types and the respective locations for the respective first visual element and the respective second visual element within the frame, a relationship between the respective first visual element and the respective second visual element,
• generating a respective visual prompt comprising a textual description of the relationship between the respective first visual element and the respective second visual element within the frame, and
• generating, based on at least one of an audio-to-text conversion of audio content associated with the frame or classification of aural elements of the audio content, a respective audio prompt comprising a textual description of the audio content associated with the frame;
• and
• receiving, based on (i) the respective visual prompt and the respective audio prompt for each frame of the plurality of frames and (ii) an additional prompt for each frame generated from at least one of metadata associated with the content item, closed captioning data associated with the content item, or audio descriptive data associated with the content item input to a large language model (LLM) trained to output descriptive information for content items, a description of the content item.