Deep reinforcement learning framework for characterizing video content

A system for performing sequence level prediction of a video scene, comprising:

• a processor;
• a memory coupled to the processor;
• and
• processor-executable instructions embodied in the memory, the instructions being configured to cause the processor to perform the following when executed:
• represent video information in a video scene as a sequence of features depicted each frame;
• represent an environment state for each time step t corresponding to each frame by the video information for time step t and predicted affective information from a previous time step t−1
• take an action A(t) with an agent controlled by a machine learning algorithm for the frame at step t, wherein an output of the action A(t) represents affective label prediction for the frame at the time step t;
• transform a pool of predicted actions to a predicted affective history at a next time step t+1;
• include the predictive affective history as part of the environment state for the next time step t+1;
• and
• generate a reward R on predicted actions up to the current time step t, by comparing them against corresponding annotated movie scene affective labels.

A non-transitory computer-readable medium having executable instructions embodied therein, the instructions being configured cause a computer to perform the following when executed:

• represent video information in a video scene as a sequence of features depicted each frame;
• represent an environment state for each time step t corresponding to each frame by the video information for time step t and predicted affective information from a previous time step t−1
• take an action A(t) with an agent controlled by a machine learning algorithm for the frame at step t, wherein an output of the action A(t) represents affective label prediction for the frame at the time step t;
• transform a pool of predicted actions to a predicted affective history at a next time step t+1;
• include the predictive affective history as part of the environment state for the next time step t+1;
• and
• generate a reward R on predicted actions up to the current time step t, by comparing them against corresponding annotated movie scene affective labels.