Gaze-aware human activity detection & anticipation
AssigneeHonda Motor Co., Ltd.
InventorsChenyi KUANG, Nakul Agarwal
Life of this patent
Estimating the term…
Abstract
The disclosure provides systems/methods of predicting future actions from a video. The disclosed systems and methods can use a video of human interactions with an object as input to predict future human actions. The disclosed systems and methods jointly detect the gaze of the human in the video and the action (or human-object interactions (HOI)) of the human in the video to predict a future gaze. The detected gaze and action, as well as the predicted future gaze, can be used to predict future actions (or HOIs) of the human in the video.
Claims20
A computer-implemented method of predicting future actions from a video, comprising:
- embedding video frames of the video as a feature encoding representing a human-object pair that is salient in a corresponding video frame;
- applying a gaze detection module to the embedded feature encoding to predict a gaze fixation heatmap;
- inputting the predicted gaze fixation heatmap and the embedded feature encoding into an action detection module to predict, for the video, a classification probability vector for human actions and to output, for the video, updated human-object interaction features;
- inputting the predicted gaze fixation heatmap and the embedded feature encoding into a gaze anticipation module to predict, for the video, future gazes and outputting future gaze encodings;
- and
- inputting the future gaze encodings and the updated human-object interaction features into an action anticipation module to predict, for the video, future actions.
A system for predicting future actions from a video, comprising:
- one or more computers and one or more storage devices storing instructions that are executable by the one or more computers to:
- embed video frames of the video as a feature encoding representing a human-object pair that is salient in a corresponding video frame;
- apply a gaze detection module to the embedded feature encoding to predict a gaze fixation heatmap;
- input the predicted gaze fixation heatmap and the embedded feature encoding into an action detection module to predict, for the video, a classification probability vector for human actions and to output, for the video, updated human-object interaction features;
- input the predicted gaze fixation heatmap and the embedded feature encoding into a gaze anticipation module to predict, for the video, future gazes and output future gaze encodings;
- and
- input the future gaze encodings and the updated human-object interaction features into an action anticipation module to predict, for the video, future actions.
A non-transitory computer-readable medium storing software comprising instructions that are executable by one or more computers to predict future actions from a video by:
- embedding video frames of the video as a feature encoding representing a human-object pair that is salient in a corresponding video frame;
- applying a gaze detection module to the embedded feature encoding to predict a gaze fixation heatmap;
- inputting the predicted gaze fixation heatmap and the embedded feature encoding into an action detection module to predict, for the video, a classification probability vector for human actions and to output, for the video, updated human-object interaction features;
- inputting the predicted gaze fixation heatmap and the embedded feature encoding into a gaze anticipation module to predict, for the video, future gazes and outputting future gaze encodings;
- and
- inputting the future gaze encodings and the updated human-object interaction features into an action anticipation module to predict, for the video, future actions.
The non-transitory computer-readable medium of 15, wherein the instructions are further executable by the one or more computers to:
- use the predicted gaze fixation heatmap and object bounding boxes for video frames corresponding to the heatmaps to generate a gaze-conditioned score matrix for each video frame;
- and
- apply the gaze-conditioned score matrix as an attention bias in a Multi-Head Self-Attention (MHSA) layer of a temporal transformer layer of an action detection module to predict, for the video, the classification probability vector for human actions.
Dependencies are inferred from the claim text.