Action classification in video clips using attention-based neural networks

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining a feature representation of a video clip comprising a key video frame from a video and one or more context video frames from the video;
• obtaining data specifying a plurality of candidate agent bounding boxes in the key video frame, wherein each candidate agent bounding box is an initial estimate of a portion of the key video frame that depicts an agent and
• for each candidate agent bounding box:
• processing the feature representation through an action transformer neural network, wherein the action transformer neural network comprises:
• a stack of action transformer layers configured to process the feature representation to generate a final query feature vector for the candidate agent bounding box, wherein each action transformer layer is configured to:
• for each of one or more attention units:
• receive input query features for the action transformer layer,
• generate, from the feature representation, key features,
• generate, from the feature representation, value features,
• apply an attention mechanism to the input query features, the key features, and the value features to generate initial updated query features;
• and
• generate output query features from the initial updated query features, wherein:
• the input query features for the first action transformer layer in the stack are features corresponding to the candidate agent bounding box in the feature representation,
• the input query features for each action transformer layer in the stack other than the first action transformer layer are generated based on the output query features for each attention unit in the preceding action transformer layer in the stack, and
• the final query features are generated based on the output query features for each attention unit in the last action transformer layer in the stack;
• and
• one or more classification output layers configured to process a final feature vector composed of the final query features to generate a respective classification score for each action in a set of possible actions that represents a likelihood that the person depicted in the candidate agent bounding box is performing the action in the video clip.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining a feature representation of a video clip comprising a key video frame from a video and one or more context video frames from the video;
• obtaining data specifying a plurality of candidate agent bounding boxes in the key video frame, wherein each candidate agent bounding box is an initial estimate of a portion of the key video frame that depicts an agent and
• for each candidate agent bounding box:
• processing the feature representation through an action transformer neural network, wherein the action transformer neural network comprises:
• a stack of action transformer layers configured to process the feature representation to generate a final query feature vector for the candidate agent bounding box, wherein each action transformer layer is configured to:
• for each of one or more attention units:
• receive input query features for the action transformer layer,
• generate, from the feature representation, key features,
• generate, from the feature representation, value features,
• apply an attention mechanism to the input query features, the key features, and the value features to generate initial updated query features;
• and
• generate output query features from the initial updated query features, wherein:
• the input query features for the first action transformer layer in the stack are features corresponding to the candidate agent bounding box in the feature representation,
• the input query features for each action transformer layer in the stack other than the first action transformer layer are generated based on the output query features for each attention unit in the preceding action transformer layer in the stack, and
• the final query features are generated based on the output query features for each attention unit in the last action transformer layer in the stack;
• and
• one or more classification output layers configured to process a final feature vector composed of the final query features to generate a respective classification score for each action in a set of possible actions that represents a likelihood that the person depicted in the candidate agent bounding box is performing the action in the video clip.