Multi-modal representation based event localization

An artificial neural network (ANN), comprising:

• a processor;
• a memory coupled with the processor;
• and
• instructions stored in the memory and operable, when executed by the processor, to cause the ANN:
• to determine, at a first stage of a multi-stage cross-attention model of the ANN, a first cross-correlation between a first representation of each modality of a plurality of modalities associated with a sequence of inputs;
• to determine, at each second stage of one or more second stages of the multi-stage cross-attention model, a second cross-correlation between first attended representations of each modality, a first attended representation of a first modality of the plurality of modalities based on the first cross-correlation and the first representation of the first modality, and the first attended representation of a second modality of the plurality of modalities based on the first cross-correlation and the first representation of the second modality;
• to generate a concatenated feature representation associated with a final second stage of the one or more second stages based on the second cross-correlation associated with the final second stage, the first attended representation of each modality, and the first representation of each modality;
• to determine a probability distribution between a set of background actions and a set of foreground actions from the concatenated feature representation;
• and
• to localize an action in the sequence of inputs based on the probability distribution.

A non-transitory computer-readable medium having program code recorded thereon for an artificial neural network (ANN), the program code executed by a processor and comprising:

• program code to determine, at a first stage of a multi-stage cross-attention model of the ANN, a first cross-correlation between a first representation of each modality of a plurality of modalities associated with a sequence of inputs;
• program code to determine, at each second stage of one or more second stages of the multi-stage cross-attention model, a second cross-correlation between first attended representations of each modality, a first attended representation of a first modality of the plurality of modalities based on the first cross-correlation and the first representation of the first modality, and the first attended representation of a second modality of the plurality of modalities based on the first cross-correlation and the first representation of the second modality;
• program code to generate a concatenated feature representation associated with a final second stage of the one or more second stages based on the second cross-correlation associated with the final second stage, the first attended representation of each modality, and the first representation of each modality;
• program code to determine a probability distribution between a set of background actions and a set of foreground actions from the concatenated feature representation;
• and
• program code to localize an action in the sequence of inputs based on the probability distribution.

An artificial neural network (ANN), comprising:

• means for determining, at a first stage of a multi-stage cross-attention model of the ANN, a first cross-correlation between a first representation of each modality of a plurality of modalities associated with a sequence of inputs;
• means for determining, at each second stage of one or more second stages of the multi-stage cross-attention model, a second cross-correlation between first attended representations of each modality,
• a first attended representation of a first modality of the plurality of modalities based on the first cross-correlation and the first representation of the first modality, and
• the first attended representation of a second modality of the plurality of modalities based on the first cross-correlation and the first representation of the second modality;
• means for generating a concatenated feature representation associated with a final second stage of the one or more second stages based on the second cross-correlation associated with the final second stage, the first attended representation of each modality, and the first representation of each modality;
• means for determining a probability distribution between a set of background actions and a set of foreground actions from the concatenated feature representation;
• and
• means for localizing an action in the sequence of inputs based on the probability distribution.