Training audio encoder neural networks using denoising losses

A method performed by one or more computers, the method comprising:

• receiving a new input, the new input comprising a spectrogram representing a new audio signal;
• processing the new input to generate an encoded representation of the new input, comprising processing the spectrogram using an audio encoder neural network to generate an encoded representation of the new audio signal, wherein the audio encoder neural network has been trained jointly with a decoder neural network, wherein the training comprises denoising a noisy segment using the decoder, wherein the denoising is conditioned on an encoded representation of a corresponding masked segment, the encoded representation being generated by the audio encoder neural network;
• and
• processing the encoded representation of the new input using a downstream neural network to perform a downstream task on the new input.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations, the operations comprising:

• obtaining a spectrogram representing an audio signal, the spectrogram comprising a plurality of segments;
• generating, from the spectrogram, a masked spectrogram, comprising generating a masked segment corresponding to a particular segment of the plurality of segments;
• processing the masked spectrogram using an audio encoder neural network to generate an encoded representation of the audio signal that comprises a respective encoded representation of the particular segment;
• generating, from the particular segment, a noisy segment;
• and
• processing the respective encoded representation of the particular segment and the noisy segment using a decoder neural network to generate a reconstruction of the particular segment;
• and
• training the decoder neural network and the audio encoder neural network on a loss function that measures a respective error between the particular segment and the reconstruction of the particular segment generated from the noisy segment.