Generating audio using auto-regressive generative neural networks
AssigneeGoogle Inc.
InventorsNeil Zeghidour, David Grangier, Marco Tagliasacchi, Raphaël Marinier, Olivier Teboul, Zalán Borsos
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a prediction of an audio signal. One of the methods includes receiving a request to generate an audio signal; obtaining a semantic representation of the audio signal; generating, using one or more generative neural networks and conditioned on at least the semantic representation, an acoustic representation of the audio signal; and processing at least the acoustic representation using a decoder neural network to generate the prediction of the audio signal.
Claims32
A computer-implemented method for generating an acoustic representation of an audio signal, the method comprising:
- receiving a request to generate an audio signal having a respective audio sample at each of a plurality of output time steps spanning a time window;
- obtaining a semantic representation of the audio signal that specifies a respective semantic token at each of a plurality of first time steps spanning the time window;
- and
- generating, using one or more generative neural networks and conditioned on at least the semantic representation, the acoustic representation of the audio signal, the acoustic representation specifying a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning the time window, wherein the set of one or more respective acoustic tokens at each of the plurality of second time steps comprises a plurality of acoustic tokens that collectively represent a prediction of an output of a residual vector quantization applied to an embedding that represents acoustic properties of the audio signal at the second time step,
- the residual vector quantization encodes the embedding using a hierarchy of a plurality of vector quantizers that each generate a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, wherein the hierarchy comprises one or more coarse vector quantizers and one or more fine vector quantizers, and
- the set of acoustic tokens at each second time step comprises, for each vector quantizer, a respective acoustic token selected from the vocabulary for the vector quantizer.
A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for generating an acoustic representation of an audio signal, wherein the operations comprise:
- receiving a request to generate an audio signal having a respective audio sample at each of a plurality of output time steps spanning a time window;
- obtaining a semantic representation of the audio signal that specifies a respective semantic token at each of a plurality of first time steps spanning the time window;
- and
- generating, using one or more generative neural networks and conditioned on at least the semantic representation, the acoustic representation of the audio signal, the acoustic representation specifying a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning the time window, wherein the set of one or more respective acoustic tokens at each of the plurality of second time steps comprises a plurality of acoustic tokens that collectively represent a prediction of an output of a residual vector quantization applied to an embedding that represents acoustic properties of the audio signal at the second time step,
- the residual vector quantization encodes the embedding using a hierarchy of a plurality of vector quantizers that each generate a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, wherein the hierarchy comprises one or more coarse vector quantizers and one or more fine vector quantizers, and
- the set of acoustic tokens at each second time step comprises, for each vector quantizer, a respective acoustic token selected from the vocabulary for the vector quantizer.
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for generating an acoustic representation of an audio signal, wherein the operations comprise:
- receiving a request to generate an audio signal having a respective audio sample at each of a plurality of output time steps spanning a time window;
- obtaining a semantic representation of the audio signal that specifies a respective semantic token at each of a plurality of first time steps spanning the time window;
- and
- generating, using one or more generative neural networks and conditioned on at least the semantic representation, the acoustic representation of the audio signal, the acoustic representation specifying a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning the time window, wherein the set of one or more respective acoustic tokens at each of the plurality of second time steps comprises a plurality of acoustic tokens that collectively represent a prediction of an output of a residual vector quantization applied to an embedding that represents acoustic properties of the audio signal at the second time step,
- the residual vector quantization encodes the embedding using a hierarchy of a plurality of vector quantizers that each generate a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, wherein the hierarchy comprises one or more coarse vector quantizers and one or more fine vector quantizers, and
- the set of acoustic tokens at each second time step comprises, for each vector quantizer, a respective acoustic token selected from the vocabulary for the vector quantizer.
Dependencies are inferred from the claim text.