Engagement prediction network based on video content

A method of predicting user engagement with video content, comprising:

• splitting a video into a set of clips, each containing a plurality of frames;
• extracting one or more per-frame features associated with one or more of the plurality of frames in each clip in the set of clips using one or more networks;
• extracting one or more per-clip features associated with each clip in the set of clips using one or more models;
• encoding one or more text features associated with each video using one or more encoders;
• generating a set of per-frame feature results associated with the one or more per-frame features using a per-frame visual processing module comprising one or more per-frame visual MLP;
• generating a set of per-clip feature results associated with the one or more per-clip features using a per-clip visual processing module;
• generating a set of text feature results associated with the one or more text features using a textual processing module;
• generating a set of fused features {O} using a fusion module comprising a fusion tool, wherein the set of fused features {O} is based on the set of per-frame feature results, the set of per-clip feature results, and the set of text feature results;
• generating a set of temporal aggregated features {H} based on the set of fused features {O} using a temporal aggregation module comprising a multi-layer self-attention architecture;
• jointly generating a predicted normalized action watch percentage and a predicted engagement continuation rate based on the set of temporal aggregated features {H} using a joint prediction module comprising two MLP layers;
• and
• generating a recommendation associated with the video using a recommendation system, wherein the recommendation is based on the predicted normalized action watch percentage and the predicted engagement continuation rate.

A non-transitory computer-readable medium storing program code comprising instructions configured to cause an electronic processor, upon execution of the instructions, to:

• split a video into a set of clips, each containing a plurality of frames;
• extract one or more semantic features from one or more of the plurality of frames in each clip in the set of clips using an image classification network;
• extract one or more distortion features from one or more of the plurality of frames in each clip in the set of clips using a distortion recognition network;
• extract one or more visual captioning features associated with each clip in the set of clips using a visual captioning model;
• extract one or more aesthetic features associated with each clip in the set of clips using a video quality evaluator;
• extract one or more action recognition features associated with each clip in the set of clips using an action recognition network;
• encode a captioning text associated with the video using a caption text encoder;
• generate one or more background sound classes based on a background sound associated with the video;
• encode the one or more background sound classes using a sound class text encoder;
• encode a title and a short description associated with the video using a descriptive text encoder;
• generate a set of visual feature results using a per-frame visual processing module comprising one or more per-frame visual multi-layer perceptrons (MLP), wherein the set of visual feature results comprises a semantic feature result based on the one or more semantic features, and a distortion feature result based on the one or more distortion features;
• generate, using a per-clip visual processing module comprising one or more per-clip visual MLPs, a visual captioning feature result based on the one or more visual captioning features, an aesthetic feature result based on the one or more aesthetic features, an action recognition feature result based on the one or more action recognition features, a captioning text result based on the captioning text, a background sound class result based on the one or more background sound classes, a title result based on the title, and a short description result based on the short description;
• generate a set of fused features {O} using a fusion module comprising a fusion tool, wherein the set of fused features {O} is based on the set of visual feature results, the visual captioning feature result, the aesthetic feature result, the action recognition feature result, the captioning text result, the background sound class result, the title result, and the short description result;
• generate a set of temporal aggregated features {H} using a temporal aggregation module comprising a multi-layer self-attention architecture, wherein the set of temporal aggregated features {H} is based on the set of fused features {O};
• jointly generate a predicted normalized action watch percentage and a predicted engagement continuation rate using a joint prediction module comprising two MLP layers, wherein the predicted normalized action watch percentage and the predicted engagement continuation rate is based on the set of temporal aggregated features {H};
• and
• generate a recommendation associated with the video using a recommendation system, wherein the recommendation is based on the predicted normalized action watch percentage and the predicted engagement continuation rate.