Compressing audio waveforms using neural networks and vector quantizers

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• receiving an audio waveform that comprises a respective audio sample for each of a plurality of time steps;
• processing the audio waveform using an encoder neural network to generate a plurality of feature vectors representing the audio waveform;
• generating a respective coded representation of each of the plurality of feature vectors using a plurality of vector quantizers that are each associated with a respective codebook of code vectors, wherein the respective coded representation of each feature vector identifies a plurality of code vectors, including a respective code vector from the codebook of each vector quantizer, that define a quantized representation of the feature vector,
• wherein the plurality of vector quantizers are ordered in a sequence, and wherein for each of the plurality of feature vectors, generating the coded representation of the feature vector comprises:
• for a first vector quantizer in the sequence of vector quantizers:
• receiving the feature vector;
• identifying, based on the feature vector, a respective code vector from the codebook of the vector quantizer to represent the feature vector;
• and
• determining a current residual vector based on an error between:
• (i) the feature vector, and (ii) the code vector that represents the feature vector,
• wherein the coded representation of the feature vector identifies the code vector that represents the feature vector;
• and
• generating a compressed representation of the audio waveform by compressing the respective coded representation of each of the plurality of feature vectors.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• receiving an audio waveform that comprises a respective audio sample for each of a plurality of time steps;
• processing the audio waveform using an encoder neural network to generate a plurality of feature vectors representing the audio waveform;
• generating a respective coded representation of each of the plurality of feature vectors using a plurality of vector quantizers that are each associated with a respective codebook of code vectors, wherein the respective coded representation of each feature vector identifies a plurality of code vectors, including a respective code vector from the codebook of each vector quantizer, that define a quantized representation of the feature vector,
• wherein the plurality of vector quantizers are ordered in a sequence, and wherein for each of the plurality of feature vectors, generating the coded representation of the feature vector comprises:
• for a first vector quantizer in the sequence of vector quantizers:
• receiving the feature vector;
• identifying, based on the feature vector, a respective code vector from the codebook of the vector quantizer to represent the feature vector;
• and
• determining a current residual vector based on an error between:
• (i) the feature vector, and (ii) the code vector that represents the feature vector,
• wherein the coded representation of the feature vector identifies the code vector that represents the feature vector;
• and
• generating a compressed representation of the audio waveform by compressing the respective coded representation of each of the plurality of feature vectors.

A method performed by one or more computers, the method comprising:

• receiving an audio waveform that comprises a respective audio sample for each of a plurality of time steps;
• processing the audio waveform using an encoder neural network to generate a plurality of feature vectors representing the audio waveform;
• generating a respective coded representation of each of the plurality of feature vectors using a plurality of vector quantizers that are each associated with a respective codebook of code vectors, wherein the respective coded representation of each feature vector identifies a plurality of code vectors, including a respective code vector from the codebook of each vector quantizer, that define a quantized representation of the feature vector;
• and
• generating a compressed representation of the audio waveform by compressing the respective coded representation of each of the plurality of feature vectors,
• wherein the encoder neural network and the codebooks of the plurality of vector quantizers are jointly trained along with a decoder neural network, and wherein the decoder neural network is configured to:
• receive a respective quantized representation of each of a plurality of feature vectors representing an input audio waveform that are generated using the encoder neural network and the plurality of vector quantizers;
• and
• process the quantized representations of the feature vectors representing the input audio waveform to generate an output audio waveform, and
• wherein the training comprises:
• obtaining a plurality of training examples that each include:
• (i) a respective input audio waveform, and (ii) a corresponding target audio waveform;
• processing the respective input audio waveform from each training example using the encoder neural network, a plurality of vector quantizers from a sequence of vector quantizers, and the decoder neural network to generate an output audio waveform that is an estimate of the corresponding target audio waveform;
• determining gradients of an objective function that depends on the respective output and target waveforms for each training example;
• and
• using the gradients of the objective function to update one or more of:
• a set of encoder neural network parameters, a set of decoder neural network parameters, or the codebooks of the plurality of vector quantizers.