Audio generation and augmentation using generative neural networks
AssigneeNVIDIA Corporation
InventorsShijia Liao, Shiyi Lan, Arun George Zachariah, Subhashree Radhakrishnan
Life of this patent
Estimating the term…
Abstract
Approaches presented herein include an audio generation system that incorporates a two-part generator having an encoder and decoder structure. An input mel-spectrogram is provided to the encoder to generate embeddings for the decoder to upsample and then produce one or more waveforms. During training, a discriminator may be used to evaluate the one or more waveforms to update weights of the encoder and/or the decoder. To conserve memory consumption, gradient checkpoints associated with the decoder may be deleted upon generation of the outputs and then, during backpropagation, gradients may be recomputed.
Claims20
A processor, comprising:
- one or more circuits to:
- upsample a first representation of first audio data to generate a second representation of the first audio data at a higher resolution than the first representation;
- generate second audio data at the higher resolution;
- and
- update one or more weights for a decoder configured to generate the second audio data using one or more recomputed gradients based on the first representation and the second representation.
A computer-implemented method, comprising:
- generating, using an encoder, a first representation of an input audio, the first representation in a frequency domain;
- upsampling, using a decoder, the first representation to generate a second representation of the input audio at a different resolution than the first representation;
- generating, using the decoder and based on the second representation, an output waveform;
- determining a quality of the output waveform using one or more discriminators;
- and
- updating one or more weights for the decoder based on a gradient computed based on an inference between the first representation and the second representation.
A system, comprising:
- one or more processors to generate audio data based at least on processing, using one or more neural networks, an upsampled representation of a mel-spectrogram input, the upsampled representation of the mel-spectrogram input being generated from an embedding obtained using an encoder with an extended context window.
Dependencies are inferred from the claim text.