Representation learning from video with spatial audio

A computer-implemented method for training a network to interpret audio-visual spatial correspondence, the method comprising:

• obtaining, by a preprocessing subsystem, a plurality of audio-visual samples;
• extracting, by the preprocessing subsystem for each audio-visual sample, a plurality of audio channels and a plurality of video frames;
• misaligning, by the preprocessing subsystem, an audio-visual spatial relationship in each of a first subset of the plurality of audio-visual samples;
• and for each of the plurality of audio-visual samples:
• calculating, with an audio subnetwork of the network, an audio feature vector for the respective audio-visual sample based on the plurality of audio channels of the respective audio-visual sample, wherein calculating the audio feature vector for the respective audio-visual sample comprises receiving, at the audio subnetwork, a first audio representation representing a first audio channel of the plurality of audio channels stacked with a second audio representation representing a second audio channel of the plurality of audio channels;
• calculating, with a visual subnetwork of the network, a visual feature vector for the respective audio-visual sample based on the plurality of video frames of the respective audio-visual sample;
• merging, with a merging subsystem of the network, the audio feature vector with the visual feature vector over a time domain to generate an audio-visual vector;
• classifying, with a pretext subnetwork of the network, the audio-visual vector into one of a first subgroup or a second subgroup, wherein the first subgroup classification indicates the audio-visual spatial relationship is misaligned in the respective audio-visual sample;
• and adjusting, by the network, parameters of the audio subnetwork, the visual subnetwork, and the pretext subnetwork based on a loss calculated based on classification of the audio-visual vector.