# US 2021/0342670 A1 · Processing sequences using convolutional neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342670 A1
- Document ID: US20210342670A1
- Status: Published application
- Kind: A1
- Application number: 17/375,250
- Filed: 2021-07-14
- Published: 2021-11-04
- Earliest claimed: 2016-09-06
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Aaron Gerard Antonius van den Oord; Sander Etienne Lea Dieleman; Nal Emmerich Kalchbrenner; Karen Simonyan; Oriol Vinyals; Lasse Espeholt
- Web page: https://patentgrep.com/patent/US20210342670A1

## CPC

- G10L 15/16: using artificial neural networks
- G06F 40/279: Recognition of textual entities
- G06F 40/44: Statistical methods, e.g. probability models
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L 13/086: {Detection of language}
- G10L 25/30: using neural networks
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G10H 2250/311: Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing sequences using convolutional neural networks. One of the methods includes, for each of the time steps: providing a current sequence of audio data as input to a convolutional subnetwork, wherein the current sequence comprises the respective audio sample at each time step that precedes the time step in the output sequence, and wherein the convolutional subnetwork is configured to process the current sequence of audio data to generate an alternative representation for the time step; and providing the alternative representation for the time step as input to an output layer, wherein the output layer is configured to: process the alternative representation to generate an output that defines a score distribution over a plurality of possible audio samples for the time step.

## Claims

### Claim 1

A neural network system implemented by one or more computers,
wherein the neural network system is configured to perform a sequence processing task by processing an input sequence of data elements comprising a plurality of data elements to generate a neural network output which characterizes the input sequence, and
wherein the neural network system comprises:
a convolutional subnetwork comprising a plurality of convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of data elements:
receive a current input sequence comprising the data element and the data elements that precede the data element in the input sequence, and
process the current input sequence to generate an alternative representation for the data element, wherein the convolutional neural network layers are causal convolutional neural network layers and the alternative representation for the data element does not depend on any data elements that follow the data element in the input sequence; and
an output subnetwork, wherein the output subnetwork is configured to receive the alternative representations of the data elements and to process the alternative representations to generate the neural network output characterizing the input sequence, wherein the neural network output comprises a plurality of sets of scores, wherein each set of scores includes a respective score for each of a plurality of possible outputs; and
a subsystem configured to select outputs in accordance with the sets of scores to generate an output sequence comprising a plurality of outputs;
wherein:
the neural network system is configured to perform speech recognition, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of graphemes that is a grapheme representation of words spoken in the audio data input sequence; or
the neural network system is configured to perform speech recognition, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of phonemes that is a phoneme representation of words spoken in the audio data input sequence; or
the neural network system is configured to perform speech recognition, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of words in a first natural language that represents worked spoken in the audio data input sequence; or
the neural network system is configured to perform a language modeling task, the input sequence is a sequence of phonemes, and the output sequence is a sequence of words in a natural language that represents the input sequence; or
the neural network system is configured to perform a language modeling task, the input sequence is a sequence of graphemes, and the output sequence is a sequence of words in a natural language that represents the input sequence; or
the neural network system is configured to perform machine translation, the input sequence is a sequence of words in a first natural language, and the output sequence is a sequence of words in a second natural language that is a translation of the input sequence into the second natural language; or
the neural network system is configured to perform a natural language processing task, the input sequence is a sequence of words in a natural language, and the output sequence is a sequence that classifies the words in in the input sequence according to a kind of output required by the natural language processing task; or
the neural network system is configured to perform a compression task, the input sequence is a sequence of data elements drawn from an input sample, and the output sequence is a compressed or encoded sample; or
the neural network system is configured to perform a medical analysis task, the input sequence is a sequence of health data elements for a particular patient, and the output sequence includes data that characterizes health of the patient or predicts future health of the patient.

### Claim 2 (depends on 1)

The neural network system of claim 1, wherein the output subnetwork comprises:
a mean pooling layer configured to:
receive a subset of the alternative representations, and
perform mean pooling on the subset of alternative representations to generate a mean pooled representation for the subset; and
one or more additional neural network layers configured to, for each subset of the alternative representations, receive the mean pooled representation for the subset and to generate a set of scores for a position in the output sequence corresponding to the subset.

### Claim 3 (depends on 1)

The neural network system of claim 1, further comprising training the neural network system using a loss function that measures, for each position in the output sequence, a respective classification accuracy of a set of scores generated by the output subnetwork for the position in the output sequence.

### Claim 4 (depends on 1)

The neural network system of claim 1, wherein the convolutional subnetwork comprises a plurality of dilated convolutional neural network layers.

### Claim 5 (depends on 4)

The neural network system of claim 4, wherein the dilated convolutional neural network layers include multiple blocks of dilated convolutional neural network layers, wherein each block comprises multiple dilated convolutional neural network layers with increasing dilation.

### Claim 6 (depends on 4)

The neural network system of claim 4, wherein one or more of the dilated convolutional neural network layers have gated activation units.

### Claim 7 (depends on 1)

The neural network system of claim 1, wherein the convolutional subnetwork comprises residual connections.

### Claim 8 (depends on 1)

The neural network system of claim 1, wherein the convolutional subnetwork comprises skip connections.

### Claim 9

A method of performing a sequence processing task by generating a neural network output from an input sequence of data elements comprising a plurality of data elements, wherein the neural network output which characterizes the input sequence, the method comprising:
for each of the plurality of data elements, providing a current input sequence that comprises the data element and the data elements preceding the data element in the input sequence to a convolutional subnetwork comprising a plurality of convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of data elements:
receive the current input sequence for the data element, and
process the current input sequence to generate an alternative representation for the data element, wherein the convolutional neural network layers are causal convolutional neural network layers and the alternative representation for the data element does not depend on any data elements that follow the data element in the input sequence; and
providing the alternative representations to an output subnetwork, wherein the output subnetwork is configured to receive the alternative representations and to process the alternative representations to generate the neural network output characterizing the input sequence, wherein the neural network output comprises a plurality of sets of scores, wherein each set of scores includes a respective score for each of a plurality of possible outputs; and
selecting outputs in accordance with the sets of scores to generate an output sequence comprising a plurality of outputs;
wherein:
the sequence processing task is a speech recognition task, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of graphemes that is a grapheme representation of words spoken in the audio data input sequence; or
the sequence processing task is a speech recognition task, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of phonemes that is a phoneme representation of words spoken in the audio data input sequence; or
the sequence processing task is a speech recognition task, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of words in a first natural language that represents worked spoken in the audio data input sequence; or
the sequence processing task is a language modeling task, the input sequence is a sequence of phonemes, and the output sequence is a sequence of words in a natural language that represents the input sequence; or
the sequence processing task is a language modeling task, the input sequence is a sequence of graphemes, and the output sequence is a sequence of words in a natural language that represents the input sequence; or
the sequence processing task is a machine translation task, the input sequence is a sequence of words in a first natural language, and the output sequence is a sequence of words in a second natural language that is a translation of the input sequence into the second natural language; or
the sequence processing task is a natural language processing task, the input sequence is a sequence of words in a natural language, and the output sequence is a sequence that classifies the words in in the input sequence according to a kind of output required by the natural language processing task; or
the sequence processing task is a compression task, the input sequence is a sequence of data elements drawn from an input sample, and the output sequence is a compressed or encoded sample; or
the sequence processing task is a medical analysis task, the input sequence is a sequence of health data elements for a particular patient, and the output sequence includes data that characterizes health of the patient or predicts future health of the patient.

### Claim 10 (depends on 9)

The method of claim 9, wherein the output subnetwork comprises:
a mean pooling layer configured to:
receive a subset of the alternative representations, and
perform mean pooling on the subset of alternative representations to generate a mean pooled representation for the subset; and
one or more additional neural network layers configured to, for each subset of the alternative representations, receive the mean pooled representation for the subset and to generate a set of scores for a position in the output sequence corresponding to the subset.

### Claim 11 (depends on 9)

The method of claim 9, further comprising training the convolutional subnetwork and the output subnetwork using a loss function that measures, for each position in the output sequence, a respective classification accuracy of a set of scores generated by the output subnetwork for the position in the output sequence.

### Claim 12 (depends on 9)

The method of claim 9, wherein the convolutional subnetwork comprises a plurality of dilated convolutional neural network layers.

### Claim 13 (depends on 12)

The method of claim 12, wherein the dilated convolutional neural network layers include multiple blocks of dilated convolutional neural network layers, wherein each block comprises multiple dilated convolutional neural network layers with increasing dilation.

### Claim 14 (depends on 12)

The method of claim 12, wherein one or more of the dilated convolutional neural network layers have gated activation units.

### Claim 15 (depends on 9)

The method of claim 9, wherein the convolutional subnetwork comprises residual connections.

### Claim 16 (depends on 9)

The method of claim 9, wherein the convolutional subnetwork comprises skip connections.

### Claim 17

One or more non-transitory computer storage media encoded with instructions that when executed by one or more computers cause the one or more computers to perform operations for accomplishing a sequence processing task by generating a neural network output from an input sequence of data elements comprising a plurality of data elements, wherein the neural network output characterizes the input sequence, the operations comprising:
for each of the plurality of data elements, providing a current input sequence that comprises the data element and the data elements preceding the data element in the input sequence to a convolutional subnetwork comprising a plurality of convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of data elements:
receive the current input sequence for the data element, and
process the current input sequence to generate an alternative representation for the data element, wherein the convolutional neural network layers are causal convolutional neural network layers and the alternative representation for the data element does not depend on any data elements that follow the data element in the input sequence; and
providing the alternative representations to an output subnetwork, wherein the output subnetwork is configured to receive the alternative representations and to process the alternative representations to generate the neural network output characterizing the input sequence, wherein the neural network output comprises a plurality of sets of scores, wherein each set of scores includes a respective score for each of a plurality of possible outputs; and
selecting outputs in accordance with the sets of scores to generate an output sequence comprising a plurality of outputs;
wherein:
the sequence processing task is a speech recognition task, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of graphemes that is a grapheme representation of words spoken in the audio data input sequence; or
the sequence processing task is a speech recognition task, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of phonemes that is a phoneme representation of words spoken in the audio data input sequence; or
the sequence processing task is a speech recognition task, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of words in a first natural language that represents worked spoken in the audio data input sequence; or
the sequence processing task is a language modeling task, the input sequence is a sequence of phonemes, and the output sequence is a sequence of words in a natural language that represents the input sequence; or
the sequence processing task is a language modeling task, the input sequence is a sequence of graphemes, and the output sequence is a sequence of words in a natural language that represents the input sequence; or
the sequence processing task is a machine translation task, the input sequence is a sequence of words in a first natural language, and the output sequence is a sequence of words in a second natural language that is a translation of the input sequence into the second natural language; or
the sequence processing task is a natural language processing task, the input sequence is a sequence of words in a natural language, and the output sequence is a sequence that classifies the words in in the input sequence according to a kind of output required by the natural language processing task; or
the sequence processing task is a compression task, the input sequence is a sequence of data elements drawn from an input sample, and the output sequence is a compressed or encoded sample; or
the sequence processing task is a medical analysis task, the input sequence is a sequence of health data elements for a particular patient, and the output sequence includes data that characterizes health of the patient or predicts future health of the patient.

### Claim 18 (depends on 17)

The non-transitory computer storage media of claim 17, wherein the output subnetwork comprises:
a mean pooling layer configured to:
receive a subset of the alternative representations, and
perform mean pooling on the subset of alternative representations to generate a mean pooled representation for the subset; and
one or more additional neural network layers configured to, for each subset of the alternative representations, receive the mean pooled representation for the subset and to generate a set of scores for a position in the output sequence corresponding to the subset.

### Claim 19 (depends on 17)

The non-transitory computer storage media of claim 17, further comprising training the convolutional subnetwork and the output subnetwork using a loss function that measures, for each position in the output sequence, a respective classification accuracy of a set of scores generated by the output subnetwork for the position in the output sequence.

### Claim 20 (depends on 17)

The non-transitory computer storage media of claim 17, wherein the convolutional subnetwork comprises a plurality of dilated convolutional neural network layers.
