Method, System, and Computer-Readable Medium for Training a Captioner Model to Generate Captions for Video Content by Analyzing and Predicting Cinematic Elements

A computing system, comprising:

• one or more processors;
• and
• one or more memories, having stored thereon instructions that, when executed, cause the computing system to:
• obtain video frames of a plurality of videos and, for at least a subset of the frames, corresponding metadata that includes focal length, camera movement, object distance and framing style;
• identify shots within the plurality of videos, each shot including a continuous sequence captured without cuts;
• apply, for each frame in the subset, frame-level labels that include focal length, camera movement details, object distance from the camera, and framing style, based on the corresponding metadata;
• train a captioner model using the frame-level labels and the frames to recognize and predict cinematic elements in video content;
• generate, for each identified shot, shot-level labels by aggregating the frame-level labels across the shot, including calculating average focal length, determining predominant framing style, and smoothing camera movement data across the shot;
• deploy the trained captioner model to process and label a video database including a plurality of videos, wherein the captioner model generates metadata for previously unseen videos based on learned cinematic elements;
• and
• post-process labels generated by the captioner model to ensure consistency and accuracy, including at least one of outlier detection, confidence scoring or manual quality control.

A non-transitory computer-readable medium having stored thereon a set of instructions that, when executed, cause a computer to:

• obtain video frames of a plurality of videos and, for at least a subset of the frames, corresponding metadata that includes focal length, camera movement, object distance and framing style;
• identify shots within the plurality of videos, each shot including a continuous sequence captured without cuts;
• apply, for each frame in the subset, frame-level labels that include focal length, camera movement details, object distance from the camera, and framing style, based on the corresponding metadata;
• train a captioner model using the frame-level labels and the frames to recognize and predict cinematic elements in video content;
• generate, for each identified shot, shot-level labels by aggregating the frame-level labels across the shot, including calculating average focal length, determining predominant framing style, and smoothing camera movement data across the shot;
• deploy the trained captioner model to process and label a video database including a plurality of videos, wherein the captioner model generates metadata for previously unseen videos based on learned cinematic elements;
• and
• post-process labels generated by the captioner model to ensure consistency and accuracy, including at least one of outlier detection, confidence scoring or manual quality control.