Processing text sequences using neural networks

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

• obtaining training data for training a neural network,
• wherein the neural network is configured to receive a network input and to process the network input in accordance with a plurality of parameters of the neural network to generate a respective score distribution for each of a plurality of output positions,
• wherein the respective score distribution for each of the output positions comprises a respective score for each token in a predetermined set of tokens,
• wherein the predetermined set of tokens includes n-grams of multiple different sizes,
• wherein, for each output position, the respective score for each of the tokens in the score distribution for the output position represents a likelihood that the token is a token at the output position in an output sequence for the network input, and
• wherein the training data comprises a plurality of training inputs, and for each training input, a respective target output sequence comprising one or more words;
• for each training input:
• processing the training input using the neural network in accordance with current values of the parameters of the neural network to generate a respective score distribution for each of a plurality of output positions;
• sampling, from a plurality of possible valid decompositions of the target output sequence for the training input, a valid decomposition of the target output sequence, wherein each possible valid decomposition of the target sequence decomposes the target sequence into a different sequence of tokens from the predetermined set of tokens;
• and
• adjusting the current values of the parameters of the neural network to increase likelihoods of the tokens in the sampled valid decomposition being the tokens at the corresponding output positions in the output sequence.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining training data for training a neural network,
• wherein the neural network is configured to receive a network input and to process the network input in accordance with a plurality of parameters of the neural network to generate a respective score distribution for each of a plurality of output positions,
• wherein the respective score distribution for each of the output positions comprises a respective score for each token in a predetermined set of tokens,
• wherein the predetermined set of tokens includes n-grams of multiple different sizes,
• wherein, for each output position, the respective score for each of the tokens in the score distribution for the output position represents a likelihood that the token is a token at the output position in an output sequence for the network input, and
• wherein the training data comprises a plurality of training inputs, and for each training input, a respective target output sequence comprising one or more words;
• for each training input:
• processing the training input using the neural network in accordance with current values of the parameters of the neural network to generate a respective score distribution for each of a plurality of output positions;
• sampling, from a plurality of possible valid decompositions of the target output sequence for the training input, a valid decomposition of the target output sequence, wherein each possible valid decomposition of the target sequence decomposes the target sequence into a different sequence of tokens from the predetermined set of tokens;
• and
• adjusting the current values of the parameters of the neural network to increase likelihoods of the tokens in the sampled valid decomposition being the tokens at the corresponding output positions in the output sequence.