# US 2018/0322891 A1 · Generating audio using neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0322891 A1 - Document ID: US20180322891A1 - Status: Published application - Kind: A1 - Application number: 16/030,742 - Filed: 2018-07-09 - Published: 2018-11-08 - Earliest claimed: 2016-09-06 - Assignees: DeepMind Technologies Limited - Applicants: DeepMind Technologies Limited - Inventors: Aaron Gerard Antonius van den Oord; Sander Etienne Lea Dieleman; Nal Emmerich Kalchbrenner; Karen Simonyan; Oriol Vinyals - Web page: https://patentgrep.com/patent/US20180322891A1 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G10L 25/30: using neural networks - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/048: Activation functions - G06N 3/09: Supervised learning - G10L 13/06: Elementary speech units used in speech synthesisers; Concatenation rules - G10H 2250/311: Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating an output sequence of audio data that comprises a respective audio sample at each of a plurality of time steps. One of the methods includes, for each of the time steps: providing a current sequence of audio data as input to a convolutional subnetwork, wherein the current sequence comprises the respective audio sample at each time step that precedes the time step in the output sequence, and wherein the convolutional subnetwork is configured to process the current sequence of audio data to generate an alternative representation for the time step; and providing the alternative representation for the time step as input to an output layer, wherein the output layer is configured to: process the alternative representation to generate an output that defines a score distribution over a plurality of possible audio samples for the time step. ## Claims ### Claim 1 A neural network system implemented by one or more computers, wherein the neural network system is configured to generate an output sequence of audio data that comprises a respective audio sample at each of a plurality of time steps, and wherein the neural network system comprises: a convolutional subnetwork comprising one or more audio-processing convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of time steps: receive a current sequence of audio data that comprises the respective audio sample at each time step that precedes the time step in the output sequence, and process the current sequence of audio data to generate an alternative representation for the time step; and an output layer, wherein the output layer is configured to, for each of the plurality of time steps: receive the alternative representation for the time step, and process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible audio samples for the time step. ### Claim 2 (depends on 1) The neural network system of claim 1, wherein the neural network system further comprises: a subsystem configured to, for each of the plurality of time steps: select an audio sample at the time step in the output sequence in accordance with the score distribution for the time step. ### Claim 3 (depends on 2) The neural network system of claim 2, wherein selecting the audio value comprises: sampling from the score distribution. ### Claim 4 (depends on 2) The neural network system of claim 2, wherein selecting the audio value comprises: selecting an audio sample having a highest score according to the score distribution. ### Claim 5 (depends on 1) The neural network system of claim 1, wherein each of the plurality of time steps corresponds to a respective time in an audio waveform, and wherein the respective audio sample at each of the plurality of time steps is an amplitude value of the audio waveform at the corresponding time. ### Claim 6 (depends on 1) The neural network system of claim 1, wherein each of the plurality of time steps corresponds to a respective time in an audio waveform, and wherein the respective audio sample at each of the plurality of time steps is a compressed or a companded representation of the audio waveform at the corresponding time. ### Claim 7 (depends on 1) The neural network system of claim 1, wherein the audio-processing convolutional neural network layers are causal convolutional neural network layers. ### Claim 8 (depends on 1) The neural network system of claim 1, wherein the audio-processing convolutional neural network layers include one or more dilated convolutional neural network layers. ### Claim 9 (depends on 8) The neural network system of claim 8, wherein the audio-processing convolutional neural network layers include multiple blocks of dilated convolutional neural network layers, wherein each block comprises multiple dilated convolutional neural network layers with increasing dilation. ### Claim 10 (depends on 1) The neural network system of claim 1, wherein one or more of the audio-processing convolutional neural network layers have gated activation units. ### Claim 11 (depends on 1) The neural network system of claim 1, wherein, at each of the plurality of time steps, the alternative representation is conditioned on a neural network input. ### Claim 12 (depends on 11) The neural network system of claim 11, wherein the neural network input comprises features of a text segment, and wherein the output sequence represents a verbalization of the text segment. ### Claim 13 (depends on 12) The neural network system of claim 12, wherein the neural network input further comprises intonation pattern values. ### Claim 14 (depends on 13) The neural network system of claim 13, wherein the neural network input comprises one or more of: speaker identity information, language identity information, and speaking style information. ### Claim 15 (depends on 1) The neural network system of claim 1, wherein the output sequence represents a piece of music. ### Claim 16 (depends on 1) The neural network system of claim 1, wherein the convolutional subnetwork comprises residual connections. ### Claim 17 (depends on 1) The neural network system of claim 1, wherein the convolutional subnetwork comprises skip connections. ### Claim 18 (depends on 1) The neural network system of claim 1, wherein processing the current sequence of audio data to generate an alternative representation for the time step comprises reusing values computed for previous time steps. ### Claim 19 One or more non-transitory computer-readable storage media encoded with instructions that when executed by one or more computers cause the one or more computers to implement a neural network system, wherein the neural network system is configured to generate an output sequence of audio data that comprises a respective audio sample at each of a plurality of time steps, and wherein the neural network system comprises: a convolutional subnetwork comprising one or more audio-processing convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of time steps: receive a current sequence of audio data that comprises the respective audio sample at each time step that precedes the time step in the output sequence, and process the current sequence of audio data to generate an alternative representation for the time step; and an output layer, wherein the output layer is configured to, for each of the plurality of time steps: receive the alternative representation for the time step, and process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible audio samples for the time step. ### Claim 20 A method of generating an output sequence of audio data that comprises a respective audio sample at each of a plurality of time steps, wherein the method comprises, for each of the plurality of time steps: providing a current sequence of audio data as input to a convolutional subnetwork comprising one or more audio-processing convolutional neural network layers, wherein the current sequence comprises the respective audio sample at each time step that precedes the time step in the output sequence, and wherein the convolutional subnetwork is configured to, for each of the plurality of time steps: receive the current sequence of audio data, and process the current sequence of audio data to generate an alternative representation for the time step; and providing the alternative representation for the time step as input to an output layer, wherein the output layer is configured to, for each of the plurality of time steps: receive the alternative representation for the time step, and process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible audio samples for the time step.