Localization of narrations in image data

A computerized system for localizing language across other modalities, the computerized system comprising:

• at least one processor;
• and
• one or more computer storage media storing computer-usable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
• receiving input data partitioned into a first modality and a second modality, the first modality being image data comprising a plurality of frames and the second modality being one of audio data or text data corresponding to the image data;
• extracting a first phrase from the second modality;
• identifying, via a neural network system, a first spatiotemporal region in the first modality corresponding to the first phrase, the first spatiotemporal region comprising a first frame from the plurality of frames and a portion of pixels within the first frame, wherein the first spatiotemporal region is identified utilizing a plurality of cross-modal attention layers in the neural network system to compare features from the first modality with features from the second modality;
• and
• causing presentation, on a graphic user interface, of an indicator of the first spatiotemporal region.

A computer-implemented method for unsupervised training of neural network system, the computer-implemented method comprising:

• receiving training data, the training data comprising a first modality and a second modality, wherein the first modality comprises image data having a plurality of frames and the second modality comprises one of audio data or text data corresponding to the image data;
• extracting a set of spatiotemporal features from different spatial and temporal portions of the image data of the first modality;
• extracting a set of word features from the second modality;
• determining a spatiotemporal feature representation and a word feature representation at least by passing the set of spatiotemporal features and the set of word features through a plurality of cross-modal attention layers within the neural network system;
• computing a contrastive loss value from the spatiotemporal feature representation and the word feature representation;
• and
• adjusting weights within the neural network system based on the contrastive loss value.