Very deep convolutional neural networks for end-to-end speech recognition

A method comprising:

• receiving an input acoustic sequence representing an utterance, the input acoustic sequence comprising a respective acoustic feature representation at each of a plurality of input time steps;
• and
• processing the input acoustic sequence using an encoder neural network to generate an encoded sequence comprising a respective encoded representation at each of a plurality of time reduced time steps, the number of time reduced time steps being less than the number of input time steps,
• wherein processing the input acoustic sequence using the encoder neural network comprises:
• processing, using a time reduction subnetwork of the encoder neural network, the input acoustic sequence to generate a sequence of reduced representations comprising a respective reduced representation at each of the plurality of time reduced time steps;
• for each time reduced time step, processing, using a convolutional Long short-term memory (LSTM) subnetwork of the encoder neural network, the reduced representation at the time reduced time step to generate a convolutional LSTM output for the time step;
• and
• for each time reduced time step, processing, using a network in network subnetwork of the encoder neural network, the convolutional LSTM output at the time reduced time step to generate the encoded representation for the time reduced time step;
• and
• processing, using a decoder neural network, the encoded sequence to generate, for each position in an output sequence order, a set of sub string scores that includes a respective sub string score for each sub string in a set of sub strings.

One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• receiving an input acoustic sequence representing an utterance, the input acoustic sequence comprising a respective acoustic feature representation at each of a plurality of input time steps;
• and
• processing the input acoustic sequence using an encoder neural network to generate an encoded sequence comprising a respective encoded representation at each of a plurality of time reduced time steps, the number of time reduced time steps being less than the number of input time steps,
• wherein processing the input acoustic sequence using the encoder neural network comprises:
• processing, using a time reduction subnetwork of the encoder neural network, the input acoustic sequence to generate a sequence of reduced representations comprising a respective reduced representation at each of the plurality of time reduced time steps;
• for each time reduced time step, processing, using a convolutional Long short-term memory (LSTM) subnetwork of the encoder neural network, the reduced representation at the time reduced time step to generate a convolutional LSTM output for the time step;
• and
• for each time reduced time step, processing, using a network in network subnetwork of the encoder neural network, the convolutional LSTM output at the time reduced time step to generate the encoded representation for the time reduced time step;
• and
• processing, using a decoder neural network, the encoded sequence to generate, for each position in an output sequence order, a set of sub string scores that includes a respective sub string score for each sub string in a set of sub strings,
• wherein the network in network subnetwork comprises a plurality of bi-directional LSTM layers.