Evaluating output sequences using an auto-regressive language model neural network

A method performed by one or more computers, the method comprising:

• receiving a request for an output sequence;
• generating, using an auto-regressive language model neural network, a first candidate output sequence, wherein the first candidate output sequence comprises a respective token at each of a plurality of positions, and wherein the auto-regressive language model neural network comprises:
• a first subnetwork that is configured to, at each particular position of the plurality of positions:
• process an input comprising a plurality of input tokens to generate a respective hidden state for each of the input tokens, the input tokens comprising tokens at each position preceding the particular position in the first candidate output sequence;
• and
• an output subnetwork configured to, at each particular position of the plurality of positions:
• process the respective hidden state for the input token at the position immediately preceding the particular position to generate a respective score for each token in a vocabulary of tokens;
• and
• select one of the tokens in the vocabulary as the token at the particular position in the first candidate output sequence using the respective scores;
• processing, using one or more classifier layers, one or more of the respective hidden states that were generated by the first subnetwork by processing an input comprising a plurality of input tokens that include all of the tokens at all of the positions in the first candidate output sequence to generate a respective rating score for each output sequence criterion in a set of one or more output sequence criteria that represents a degree to which the first candidate output sequence generated by the auto-regressive language model neural network satisfies the output sequence criterion;
• and
• providing the first candidate output sequence in response to the request using the respective rating scores for the first candidate output sequence for the set of one or more output sequence criteria.

A system comprising:

• one or more computers;
• and
• one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• receiving a request for an output sequence;
• generating, using an auto-regressive language model neural network, a first candidate output sequence, wherein the first candidate output sequence comprises a respective token at each of a plurality of positions, and wherein the auto-regressive language model neural network comprises:
• a first subnetwork that is configured to, at each particular position of the plurality of positions:
• process an input comprising a plurality of input tokens to generate a respective hidden state for each of the input tokens, the input tokens comprising tokens at each position preceding the particular position in the first candidate output sequence;
• and
• an output subnetwork configured to, at each particular position of the plurality of positions:
• process the respective hidden state for the input token at the position immediately preceding the particular position to generate a respective score for each token in a vocabulary of tokens;
• and
• select one of the tokens in the vocabulary as the token at the particular position in the first candidate output sequence using the respective scores;
• processing, using one or more classifier layers, one or more of the respective hidden states that were generated by the first subnetwork by processing an input comprising a plurality of input tokens that include all of the tokens at all of the positions in the first candidate output sequence to generate a respective rating score for each output sequence criterion in a set of one or more output sequence criteria that represents a degree to which the first candidate output sequence generated by the auto-regressive language model neural network satisfies the output sequence criterion;
• and
• providing the first candidate output sequence in response to the request using the respective rating scores for the first candidate output sequence for the set of one or more output sequence criteria.

One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• receiving a request for an output sequence;
• generating, using an auto-regressive language model neural network, a first candidate output sequence, wherein the first candidate output sequence comprises a respective token at each of a plurality of positions, and wherein the auto-regressive language model neural network comprises:
• a first subnetwork that is configured to, at each particular position of the plurality of positions:
• process an input comprising a plurality of input tokens to generate a respective hidden state for each of the input tokens, the input tokens comprising tokens at each position preceding the particular position in the first candidate output sequence;
• and
• an output subnetwork configured to, at each particular position of the plurality of positions:
• process the respective hidden state for the input token at the position immediately preceding the particular position to generate a respective score for each token in a vocabulary of tokens;
• and
• select one of the tokens in the vocabulary as the token at the particular position in the first candidate output sequence using the respective scores;
• processing, using one or more classifier layers, one or more of the respective hidden states that were generated by the first subnetwork by processing an input comprising a plurality of input tokens that include all of the tokens at all of the positions in the first candidate output sequence to generate a respective rating score for each output sequence criterion in a set of one or more output sequence criteria that represents a degree to which the first candidate output sequence generated by the auto-regressive language model neural network satisfies the output sequence criterion;
• and
• providing the first candidate output sequence in response to the request using the respective rating scores for the first candidate output sequence for the set of one or more output sequence criteria.