Neural networks with linear recurrence and feedforward units

One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to implement:

• a neural network configured to perform a machine learning task, the neural network comprising a plurality of layer blocks each comprising:
• (i) a linear recurrent layer, and (ii) one or more feedforward layers,
• wherein each layer block is configured to perform operations comprising:
• receiving an input sequence for the layer block that comprises a respective input at each of a plurality of input positions;
• and
• generating an output sequence for the layer block that comprises a respective output at each of a plurality of output positions, the generating comprising:
• processing the input sequence, using the linear recurrent layer of the layer block, to generate an initial output sequence that comprises a respective initial output at each of the plurality of input positions;
• and
• processing the initial output sequence, using the one or more feedforward layers of the layer block, to generate the output sequence.

A method performed by one or more computers, the method comprising:

• receiving a network input;
• and
• processing the network input using a neural network to generate a network output for the network input, the neural network comprising a plurality of layer blocks each comprising:
• (i) a linear recurrent layer, and (ii) one or more feedforward layers,
• wherein each layer block is configured to perform operations comprising:
• receiving an input sequence for the layer block that comprises a respective input at each of a plurality of input positions;
• and
• generating an output sequence for the layer block that comprises a respective output at each of a plurality of output positions, the generating comprising:
• processing the input sequence, using the linear recurrent layer of the layer block, to generate an initial output sequence that comprises a respective initial output at each of the plurality of input positions;
• and
• processing the initial output sequence, using the one or more feedforward layers of the layer block, to generate the output sequence.