Training generative neural network systems using multiple reward models

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a generative neural network system, the generative neural network system configured to generate an output sequence by processing an input sequence using a sequence processing neural network, the operations comprising:

• obtaining a set of training examples, each training example comprising a training input sequence and a set of one or more corresponding training output sequences, wherein the obtaining comprises processing each training input sequence using the generative neural network system to generate each of the one or more corresponding training output sequences;
• and, for each of the training examples:
• processing the training example using a plurality of different reward models to generate a respective plurality of reward values;
• combining the reward values from each of the reward models to obtain an aggregated reward;
• and
• training the sequence processing neural network of the generative neural network system using an objective function determined using the aggregated reward.

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a generative neural network system, the generative neural network system configured to generate an output sequence by processing an input sequence using a sequence processing neural network, the operations comprising:
• obtaining a set of training examples, each training example comprising a training input sequence and a set of one or more corresponding training output sequences, wherein the obtaining comprises processing each training input sequence using the generative neural network system to generate each of the one or more corresponding training output sequences;
• and, for each of the training examples:
• processing the training example using a plurality of different reward models to generate a respective plurality of reward values;
• combining the reward values from each of the reward models to obtain an aggregated reward;
• and
• training the sequence processing neural network of the generative neural network system using an objective function determined using the aggregated reward.