Budget-aware method for detecting activity in video

A system for detecting activity in video, comprising:

• a computer processor executing a reinforcement learning algorithm to learn a policy to sequentially select a subset of frames of a video and classify activity within the subset of frames according to a plurality of predefined activity classifications;
• and
• a recurrent neural network that uses the policy to detect and classify activity in the video over a sequence of steps determined according to a given time and/or computation budget, including for each step of the plurality of steps:
• selecting a subset of frames of the video,
• predicting a segment from the selected subset of frames including a temporal location of the segment in the video, one or more of the plurality of predefined activity classifications associated with the segment, and a next subset of frames to select for a next step of the plurality of steps.

A non-transitory computer-readable media storing computer instructions for detecting activity in video that, when executed by one or more processors, cause the one or more processors to perform the method comprising:

• learning, through a reinforcement learning algorithm, a policy to sequentially select a subset of frames of a video and classify activity within the subset of frames according to a plurality of predefined activity classifications;
• using, by a recurrent neural network, the policy to detect and classify activity in the video over a sequence of steps determined according to a given time and/or computation budget, including for each step of the plurality of steps:
• selecting a subset of frames of the video,
• predicting a segment from the selected subset of frames including a temporal location of the segment in the video, one or more of the plurality of predefined activity classifications associated with the segment, and a next subset of frames to select for a next step of the plurality of steps.