Deep reinforcement learning framework for characterizing video content

A system for performing sequence level prediction of a video scene, comprising:

• a processor;
• a memory coupled to the processor;
• and
• processor-executable instructions embodied in the memory, the instructions being configured to cause the processor to perform the following when executed:
• a) providing one or more scene affective labels at the end of the sequence, wherein each label of the one or more scene affective labels pertains to the entire sequence of frames of data;
• b) taking an action with an agent controlled by a machine learning algorithm for a current frame of the sequence at a current time step, wherein an output of the action represents affective label prediction for the frame at the current time step;
• c) transforming a pool of actions taken up until the current time step including the action taken with the agent into a predicted affective history for a subsequent time step;
• generating a reward on predicted actions up to the current time step, by comparing the predicted actions against corresponding annotated scene affective labels.

A non-transitory computer-readable medium having executable instructions embodied therein, the instructions being configured cause a computer to perform the following when executed:

• a) providing one or more scene affective labels at the end of the sequence, wherein each label of the one or more scene affective labels pertains to the entire sequence of frames of data;
• b) taking an action with an agent controlled by a machine learning algorithm for a current frame of the sequence at a current time step, wherein an output of the action represents affective label prediction for the frame at the current time step;
• c) transforming a pool of actions taken up until the current time step including the action taken with the agent into a predicted affective history for a subsequent time step;
• generating a reward on predicted actions up to the current time step, by comparing the predicted actions against corresponding annotated scene affective labels.