Action anticipation

A computer-implemented method for action anticipation, comprising:

• generating a gaze embedding associated with a gaze, a pose embedding associated with a pose, and an object embedding associated with an object from an input set of image frames;
• generating a joint cue based on the gaze embedding and the pose embedding and a global context token based on the gaze embedding and the object embedding;
• and
• generating an anticipated human-object-interaction (HOI) based on a model, the joint cue, and the global context token, wherein the model is trained using ground truth data including HOI triplet annotations.

A system for action anticipation, comprising:

• a memory storing one or more instructions;
• and
• a processor executing one or more of the instructions stored on the memory to perform:
• generating a gaze embedding associated with a gaze, a pose embedding associated with a pose, and an object embedding associated with an object from an input set of image frames;
• generating a joint cue based on the gaze embedding and the pose embedding and a global context token based on the gaze embedding and the object embedding, wherein the joint cue represents an attention and a body configuration;
• and
• generating an anticipated human-object-interaction (HOI) based on a model, the joint cue, and the global context token, wherein the model is trained using ground truth data including HOI triplet annotations.