Determining audio and video representations using self-supervised learning

A system comprising:

• a memory component;
• and
• a processing device coupled to the memory component, the processing device to perform operations comprising:
• receiving a video signal including an audio component and a video component;
• training a first machine learning model to determine a representation of the audio component using a first loss comprising a combination of a first contrastive learning loss based on a contrastive learning task executed by a first head of the first machine learning model and first temporal loss based on a temporal learning task executed by a plurality of heads of the first machine learning model, one head of the plurality of heads of the first machine learning model captures short-term features using a single segment of the audio component and another head of the plurality of heads of the first machine learning model captures long-term features using a plurality of segments of the audio component;
• and
• training a second machine learning model to determine a representation of the video component a second loss comprising a combination of a second contrastive learning loss based on using the contrastive learning task executed by a first head of the second machine learning model and second temporal loss based on the temporal learning task executed by a plurality of heads of the second machine learning model, wherein one head of the plurality of heads of the second machine learning model captures short-term features using a single segment of the video component and another head of the plurality of heads of the second machine learning model captures long-term features using a plurality of segments of the video component.

A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

• determining a first feature vector using a first machine learning model trained to determine the first feature vector using a first loss comprising a combination of a first contrastive learning loss based on a contrastive learning task executed by a first head of the first machine learning model and first temporal loss based on a temporal learning task executed by a plurality of heads of the first machine learning model, wherein one head of the plurality of heads of the first machine learning model captures short-term features and another head of the plurality of heads of the first machine learning model captures long-term features;
• providing the first feature vector to a processing module to perform a processing task;
• and
• outputting a result of the processing task.