# US 12,190,896 B2 · Generating audio waveforms using encoder and decoder neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,190,896 B2
- Document ID: US12190896B2
- Status: Granted
- Kind: B2
- Application number: 17/856,292
- Filed: 2022-07-01
- Published: 2023-01-19
- Granted: 2025-01-07
- Earliest claimed: 2021-07-02
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Yunpeng Li; Marco Tagliasacchi; Dominik Roblek; Félix de Chaumont Quitry; Beat Gfeller; Hannah Raphaelle Muckenhirn; Victor Ungureanu; Oleg Rybakov; Karolis Misiunas; Zalán Borsos
- Web page: https://patentgrep.com/patent/US12190896B2

## CPC

- G10L 19/022: Blocking, i.e. grouping of samples in time; Choice of analysis windows; Overlap factoring
- G10L 21/02: Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08)
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G10L 21/0208: Noise filtering
- G10L 21/038: using band spreading techniques
- G10L 21/06: Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids (G10L15/26 takes precedence)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing an input audio waveform using a generator neural network to generate an output audio waveform. In one aspect, a method comprises: receiving an input audio waveform; processing the input audio waveform using an encoder neural network to generate a set of feature vectors representing the input audio waveform; and processing the set of feature vectors representing the input audio waveform using a decoder neural network to generate an output audio waveform that comprises a respective output audio sample for each of a plurality of output time steps.

## Claims

### Claim 1

A method performed by one or more computers, the method comprising:
receiving an input audio waveform that comprises a respective input audio sample for each of a plurality of input time steps;
processing the input audio waveform using an encoder neural network to generate a set of feature vectors representing the input audio waveform,
wherein the encoder neural network comprises a sequence of encoder blocks that are each configured to:
process a respective set of input feature vectors in accordance with a set of encoder block parameters to generate a set of output feature vectors, comprising down-sampling the set of input feature vectors; and
processing the set of feature vectors representing the input audio waveform using a decoder neural network to generate an output audio waveform that comprises a respective output audio sample for each of a plurality of output time steps,
wherein the decoder neural network comprises a sequence of decoder blocks that are each configured to:
process a respective set of input feature vectors in accordance with a set of decoder block parameters to generate a set of output feature vectors, comprising up-sampling the set of input feature vectors;
wherein the output audio waveform represents a version of the input audio waveform that has been filtered to include only audio from a target audio source; and
wherein the encoder neural network, the decoder neural network, or both additionally process a conditioning vector representing the target audio source.

### Claim 2 (depends on 1)

The method of claim 1, wherein each encoder block in the sequence of encoder blocks down-samples the set of input feature vectors to the encoder block using a respective strided convolution operation.

### Claim 3 (depends on 2)

The method of claim 2, wherein for each encoder block in the sequence of encoder blocks, the strided convolution operation is a one-dimensional or two-dimensional strided convolution operation.

### Claim 4 (depends on 1)

The method of claim 1, wherein for each encoder block in the sequence of encoder blocks, a dimensionality of the output feature vectors generated by the encoder block is higher than a dimensionality of the input feature vectors processed by the encoder block.

### Claim 5 (depends on 1)

The method of claim 1, wherein each decoder block in the sequence of decoder blocks up-samples the set of input feature vectors to the decoder block using a respective strided transposed convolution operation.

### Claim 6 (depends on 5)

The method of claim 5, wherein for each decoder block in the sequence of decoder blocks, the strided transposed convolution operation is a one-dimensional or two-dimensional strided transposed convolution operation.

### Claim 7 (depends on 1)

The method of claim 1, wherein for each decoder block in the sequence of decoder blocks, a dimensionality of the output feature vectors generated by the decoder block is lower than a dimensionality of the input feature vectors processed by the decoder block.

### Claim 8 (depends on 1)

The method of claim 1, wherein for each encoder block that is after a first encoder block in the sequence of encoder blocks, the set of input feature vectors to the encoder block comprises a set of output feature vectors generated by a preceding encoder block in the sequence of encoder blocks.

### Claim 9 (depends on 1)

The method of claim 1, wherein for each decoder block that is after a first decoder block in the sequence of decoder blocks, the set of input feature vectors to the decoder block comprises: (i) a set of output feature vectors of a corresponding encoder block, and (ii) a set of output feature vectors generated by a preceding decoder block in the sequence of decoder blocks.

### Claim 10 (depends on 1)

The method of claim 1, wherein the encoder neural network comprises a transform layer prior to the sequence of encoder blocks, wherein the transform layer maps the input audio waveform to an alternative representation in an alternative domain.

### Claim 11 (depends on 10)

The method of claim 10, wherein the transform layer maps the input audio waveform to an alternative representation in a time-frequency domain.

### Claim 12 (depends on 11)

The method of claim 11, wherein the transform layer implements a Fourier transform operation.

### Claim 13 (depends on 10)

The method of claim 10, wherein the decoder neural network comprises an inverse transform layer after the sequence of decoder blocks, wherein the inverse transform layer maps a representation of the output audio waveform in the alternative domain to a representation of the audio waveform in a time domain.

### Claim 14 (depends on 13)

The method of claim 13, wherein the inverse transform layer implements an inverse Fourier transform operation.

### Claim 15 (depends on 1)

The method of claim 1, wherein the encoder neural network and the decoder neural network are jointly trained, and the training comprises:
obtaining a plurality of training examples that each include: (i) a respective input audio waveform, and (ii) a corresponding target audio waveform;
processing the respective input audio waveform from each training example using the encoder neural network followed by the decoder neural network to generate an output audio waveform that is an estimate of the corresponding target audio waveform;
determining gradients of an objective function that depends on the respective output waveform and respective target waveform for each training example; and
using the gradients of the objective function to update a set of encoder neural network parameters and a set of decoder neural network parameters.

### Claim 16 (depends on 15)

The method of claim 15, wherein the training further comprises, for each training example:
processing data derived from the output audio waveform using a discriminator neural network to generate a set of one or more discriminator scores, wherein each discriminator score characterizes an estimated likelihood that the output audio waveform is an audio waveform that was generated using the encoder neural network and the decoder neural network;
wherein the objective function comprises an adversarial loss that depends on the discriminator scores generated by the discriminator neural network.

### Claim 17 (depends on 16)

The method of claim 16, wherein the data derived from the output audio waveform comprises the output audio waveform, a down-sampled version of the output audio waveform, or a Fourier-transformed version of the output audio waveform.

### Claim 18 (depends on 16)

The method of claim 16, wherein the training further comprises, for each training example:
generating a respective set of discriminator scores using each of a plurality of discriminator neural networks, wherein each discriminator neural network processes a respective version of the output audio waveform that has been down-sampled by a respective factor;
wherein the adversarial loss depends on the discriminator scores generated by the plurality of discriminator neural networks.

### Claim 19 (depends on 16)

The method of claim 16, wherein the discriminator neural network is trained to generate discriminator scores that distinguish between: (i) output audio waveforms generated using the encoder neural network and the decoder neural network, and (ii) target audio waveforms from training examples.

### Claim 20 (depends on 16)

The method of claim 16, wherein the discriminator neural network is a convolutional neural network, and wherein a number of discriminator scores in the set of discriminator scores generated by the discriminator neural network is proportional to a length of the output audio waveform.

### Claim 21 (depends on 16)

The method of claim 16, wherein the objective function comprises a reconstruction loss that, for each training example, measures an error between: (i) the output audio waveform, and (ii) the corresponding target audio waveform.

### Claim 22 (depends on 21)

The method of claim 21, wherein for each training example, the reconstruction loss measures a discrepancy between: (i) one or more intermediate output generated by the discriminator neural network by processing the output audio waveform, and (ii) one or more intermediate outputs generated by the discriminator neural network by processing the corresponding target audio waveform.

### Claim 23 (depends on 21)

The method of claim 21, wherein the objective function comprises a linear combination of the adversarial loss and the reconstruction loss.

### Claim 24 (depends on 1)

The method of claim 1, wherein each encoder block and each decoder block comprise one or more respective dilated convolutional neural network layers.

### Claim 25 (depends on 1)

The method of claim 1, wherein each encoder block and each decoder block comprise one or more respective residual blocks.

### Claim 26 (depends on 1)

The method of claim 1, wherein the output audio waveform is an enhanced version of the input audio waveform.

### Claim 27 (depends on 26)

The method of claim 26, wherein the output audio waveform is a de-noised version of the input audio waveform.

### Claim 28 (depends on 27)

The method of claim 27, wherein the encoder neural network processes both: (i) the input audio waveform, and (ii) corresponding accelerometer data.

### Claim 29 (depends on 28)

The method of claim 28, wherein the accelerometer data comprises an accelerometer waveform that is captured synchronously with the input audio waveform.

### Claim 30 (depends on 29)

The method of claim 29, wherein the accelerometer data comprises an accelerometer waveform captured by a bone conductance accelerometer mounted on a headphone.

### Claim 31 (depends on 26)

The method of claim 26, wherein the output audio waveform represents a version of the input audio waveform captured at a higher sampling frequency than the input audio waveform.

### Claim 32 (depends on 31)

The method of claim 31, wherein the encoder neural network and the decoder neural network are trained on input audio waveforms having a range of sampling frequencies.

### Claim 33 (depends on 31)

The method of claim 31, wherein the input audio waveform is generated as an output of a text-to-speech model that processes an input sequence of text to generate an audio waveform that is a verbalization of the input sequence of text.

### Claim 34 (depends on 1)

The method of claim 1, wherein the input audio waveform is captured using a target microphone, and the output audio waveform represents a version of the input audio waveform captured using a source microphone.

### Claim 35 (depends on 34)

The method of claim 34, further comprising using the input audio waveform and the corresponding output audio waveform to optimize parameters of a target microphone model that defines a mapping from audio waveforms captured using the source microphone to audio waveforms captured using the target microphone.

### Claim 36 (depends on 35)

The method of claim 35, further comprising:
processing a plurality of source audio waveforms corresponding to the source microphone using the optimized target microphone model to generate a plurality of target audio waveforms corresponding to the target microphone; and
training a machine learning model to perform an audio processing task using the target audio waveforms corresponding to the target microphone.

### Claim 37 (depends on 1)

The method of claim 1, wherein the input audio waveform is a speech waveform corresponding to a verbalization by a person, and wherein the output audio waveform represents an anonymized version of the verbalization that would be generated by a text-to-speech model.

### Claim 38 (depends on 1)

The method of claim 1, wherein the target audio source corresponds to a particular speaker.

### Claim 39 (depends on 1)

The method of claim 1, wherein the target audio source corresponds to a particular musical instrument.

### Claim 40 (depends on 1)

The method of claim 1, wherein the conditioning vector representing the target audio source is generated based on a sample audio waveform from the target audio source, wherein generating the conditioning vector comprises:
processing the sample audio waveform from the target audio source using a conditioning neural network to generate a plurality of feature vectors representing the sample audio waveform; and
generating the conditioning vector using the plurality of feature vectors representing the sample audio waveform.

### Claim 41 (depends on 40)

The method of claim 40, wherein the conditioning neural network comprises a sequence of encoder blocks that are each configured to:
process a respective set of input feature vectors in accordance with a set of encoder block parameters to generate a set of output feature vectors, comprising down-sampling the set of input feature vectors.

### Claim 42 (depends on 40)

The method of claim 40, wherein generating the conditioning vector using the plurality of feature vectors representing the sample audio waveform comprises:
pooling the plurality of feature vectors representing the sample audio waveform to generate a pooled feature vector;
determining a respective score for each of the plurality of feature vectors representing the sample audio waveform based on a similarity between: (i) the pooled feature vector, and (ii) the feature vector representing the sample audio waveform; and
generating the conditioning vector by combining the plurality of feature vectors representing the sample audio waveform using the scores.

### Claim 43 (depends on 1)

The method of claim 1, wherein the input audio waveform is a speech or music waveform.

### Claim 44 (depends on 1)

The method of claim 1, wherein the encoder neural network and the decoder neural network are implemented on a mobile device.

### Claim 45 (depends on 1)

The method of claim 1, wherein the encoder neural network and the decoder neural network are implemented on a personal digital assistant device.

### Claim 46

A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
receiving an input audio waveform that comprises a respective input audio sample for each of a plurality of input time steps;
processing the input audio waveform using an encoder neural network to generate a set of feature vectors representing the input audio waveform,
wherein the encoder neural network comprises a sequence of encoder blocks that are each configured to:
process a respective set of input feature vectors in accordance with a set of encoder block parameters to generate a set of output feature vectors, comprising down-sampling the set of input feature vectors; and
processing the set of feature vectors representing the input audio waveform using a decoder neural network to generate an output audio waveform that comprises a respective output audio sample for each of a plurality of output time steps,
wherein the decoder neural network comprises a sequence of decoder blocks that are each configured to:
process a respective set of input feature vectors in accordance with a set of decoder block parameters to generate a set of output feature vectors, comprising up-sampling the set of input feature vectors;
wherein the output audio waveform represents a version of the input audio waveform that has been filtered to include only audio from a target audio source; and
wherein the encoder neural network, the decoder neural network, or both additionally process a conditioning vector representing the target audio source.

### Claim 47

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving an input audio waveform that comprises a respective input audio sample for each of a plurality of input time steps;
processing the input audio waveform using an encoder neural network to generate a set of feature vectors representing the input audio waveform,
wherein the encoder neural network comprises a sequence of encoder blocks that are each configured to:
process a respective set of input feature vectors in accordance with a set of encoder block parameters to generate a set of output feature vectors, comprising down-sampling the set of input feature vectors; and
processing the set of feature vectors representing the input audio waveform using a decoder neural network to generate an output audio waveform that comprises a respective output audio sample for each of a plurality of output time steps,
wherein the decoder neural network comprises a sequence of decoder blocks that are each configured to:
process a respective set of input feature vectors in accordance with a set of decoder block parameters to generate a set of output feature vectors, comprising up-sampling the set of input feature vectors;
wherein the output audio waveform represents a version of the input audio waveform that has been filtered to include only audio from a target audio source; and
wherein the encoder neural network, the decoder neural network, or both additionally process a conditioning vector representing the target audio source.