Reward weighting alignment of large language models

A non-transitory computer-readable medium storing instructions that, when executed by a computing system within a cloud provider network, cause the computing system to perform operations comprising:

• populating one or more tokens for a prompt using a large language model, wherein the one or more tokens are populated in one or more beams;
• identifying one or more framework parameters for evaluating the one or more beams, the one or more framework parameters comprising:
• a reward parameter having a first criterion associated with an alignment score derived from an alignment model trained using a set of constraints;
• and
• a constraint checker comprising a second criterion as a constraint, wherein the second criterion is associated with a constraint score;
• evaluating the one or more beams based on the second criterion to identify a subset of beams of the one or more beams, wherein the subset of beams comprises each beam of the one or more beams that satisfies the second criterion;
• evaluating each beam of the subset of beams based on the first criterion;
• calculating a first criterion score for each beam of the subset of beams, wherein calculating the first criterion score comprises adjusting by a first weight;
• calculating a second criterion score for each beam of the subset of beams, wherein calculating the second criterion score comprising adjusting by a second weight;
• selecting, based on a combined preference score for each beam of the subset of beams, a reward parameter beam;
• and
• transmitting the reward parameter beam.

A system comprising:

• a data storage medium to store computer-executable instructions;
• and
• a processor in communication with the data storage medium, wherein the processor executes the computer-executable instructions to at least:
• populate one or more tokens for a prompt using a large language model, wherein the one or more tokens are populated in one or more beams;
• identify one or more framework parameters for evaluating the one or more beams comprising a first criterion and a second criterion;
• evaluate the one or more beams based on the first criterion and the second criterion, wherein the first criterion is associated with a first score and the second criterion is associated with a second score;
• calculate a first criterion score for each of the one or more beams using the first score of the first criterion adjusted by a first weight;
• calculate a second criterion score each of the one or more beams using the second score of the second criterion adjusted by a second weight;
• and
• select, based on a combined preference score for each of the one or more beams, a reward parameter beam, wherein the combined preference score is determined using the first criterion score and the second criterion score.