Single trajectory policy optimization for generative machine learning models

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• training a target generative machine learning model, the training comprising, at each of a sequence of training iterations:
• obtaining a plurality of training examples for the training iteration, wherein each training example includes:
• (i) an example prompt for the training example,
• (ii) an example data item for the training example, and
• (iii) a quality score for the training example that measures a quality of the example data item given the example prompt;
• determining, for each of the plurality of training examples for the training iteration, a likelihood of the target generative machine learning model generating the example data item by processing the example prompt for the training example;
• determining, for each of the plurality of training examples for the training iteration, an expected quality score for the training example;
• and
• training the target generative machine learning model to optimize an objective function, wherein the objective function depends on, for each training example for the training iteration, (i) the likelihood of the target generative machine learning model generating the example data item by processing the example prompt for the training example and (ii) a difference between the quality score for the training example and the expected quality score for the training example.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• training a target generative machine learning model, the training comprising, at each of a sequence of training iterations:
• obtaining a plurality of training examples for the training iteration, wherein each training example includes:
• (i) an example prompt for the training example,
• (ii) an example data item for the training example, and
• (iii) a quality score for the training example that measures a quality of the example data item given the example prompt;
• determining, for each of the plurality of training examples for the training iteration, a likelihood of the target generative machine learning model generating the example data item by processing the example prompt for the training example;
• determining, for each of the plurality of training examples for the training iteration, an expected quality score for the training example;
• and
• training the target generative machine learning model to optimize an objective function, wherein the objective function depends on, for each training example for the training iteration, (i) the likelihood of the target generative machine learning model generating the example data item by processing the example prompt for the training example and (ii) a difference between the quality score for the training example and the expected quality score for the training example.