Methods and systems for learning language-invariant audiovisual representations

A system comprising:

• at least one physical processor;
• and
• physical memory comprising computer-executable instructions that, when executed by the at least one physical processor, cause the at least one physical processor to perform acts comprising:
• generating a training set comprising, from a long-form video, a video track clip, a primary language audio track corresponding to the video track clip, and a dubbed language audio track corresponding to the video track clip, wherein the dubbed language audio track preserves non-speech audio corresponding to the video track clip and replaces spoken language with translated spoken language;
• applying, by one or more physical processors, a machine-learning model to the training set to generate:
• a video representation of the video track clip,
• a primary-language audio representation of the primary language audio track, and
• a dubbed-language audio representation of the dubbed language audio track;
• projecting, by one or more projection heads of the machine-learning model, the video representation, the primary-language audio representation, and the dubbed-language audio representation into a shared representational space;
• computing, by the one or more physical processors, a contrastive cost between modalities, the contrastive cost comprising a first cost between the projected video representation and the projected primary-language audio representation, and a second cost between the projected video representation and the projected dubbed-language audio representation, wherein the contrastive cost comprises a sum of the first cost and the second cost;
• and
• iteratively updating parameters of the machine-learning model based on the contrastive cost over multiple training epochs to train the machine-learning model to position the projected video representation within a threshold distance of each of the projected primary-language audio representation and the projected dubbed-language audio representation within the shared representational space.

A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

• generate a training set comprising, from a long-form video, a video track clip, a primary language audio track corresponding to the video track clip, a first dubbed language audio track corresponding to the video track clip, and a second dubbed language audio track corresponding to the video track clip, wherein:
• the first dubbed language audio track preserves non-speech audio corresponding to the video track clip and replaces spoken language with a first translated spoken language, and
• the second dubbed language audio track preserves non-speech audio corresponding to the video track clip and replaces spoken language with a second translated spoken language;
• apply, by one or more physical processors, a machine-learning model to the training set to generate:
• a video representation of the video track clip,
• a primary-language audio representation of the primary language audio track,
• a first dubbed-language audio representation of the first dubbed language audio track, and
• a second dubbed-language audio representation of the second dubbed language audio track;
• project, by one or more projection heads of the machine-learning model, the video representation, the primary-language audio representation, the first dubbed-language audio representation, and the second dubbed-language audio representation into a shared representational space;
• compute, by the one or more physical processors, a contrastive cost between modalities, the contrastive cost comprising a first cost between the projected video representation and the projected primary-language audio representation, a second cost between the projected video representation and the projected first dubbed-language audio representation, and a third cost between the projected video representation and the projected second dubbed-language audio representation, wherein the contrastive cost comprises a sum of the first cost, the second cost, and the third cost;
• and
• iteratively update parameters of the machine-learning model based on the contrastive cost over multiple training epochs to train the machine-learning model to position the projected video representation within a threshold distance of each of the projected primary-language audio representation, the projected first dubbed-language audio representation, and the projected second dubbed-language audio representation within the shared representational space.