Processing sequences using convolutional neural networks

One or more non-transitory computer storage media encoded with instructions that when executed by one or more computers cause the one or more computers to implement a neural network system,

• wherein the neural network system is configured to generate an output sequence of data elements that comprises a respective data element at each of a plurality of time steps, and
• wherein the neural network system comprises:
• a convolutional subnetwork comprising one or more data-processing convolutional neural network layers, wherein the data-processing convolutional neural network layers are causal convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of time steps:
• receive a current sequence of data elements that comprises the respective data element at each time step that precedes the time step in the output sequence, and
• process the current sequence of data elements to generate an alternative representation for the time step;
• and
• an output layer, wherein the output layer is configured to, for each of the plurality of time steps:
• receive the alternative representation for the time step, and
• process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible data elements for the time step.

A method of generating an output sequence of data elements that comprises a respective data element at each of a plurality of time steps,

• wherein the method comprises, for each of the plurality of time steps:
• providing a current sequence of data elements as input to a convolutional subnetwork comprising one or more data-processing convolutional neural network layers,
• wherein the data-processing convolutional neural network layers are causal convolutional neural network layers,
• wherein the current sequence comprises the respective data element at each time step that precedes the time step in the output sequence, and
• wherein the convolutional subnetwork is configured to, for each of the plurality of time steps:
• receive the current sequence of data elements, and
• process the current sequence of data elements to generate an alternative representation for the time step;
• and
• providing the alternative representation for the time step as input to an output layer, wherein the output layer is configured to, for each of the plurality of time steps:
• receive the alternative representation for the time step, and
• process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible data elements for the time step.