Alignment model as decoding-time heuristic
AssigneeAmazon Technologies, Inc.
InventorsSailik Sengupta, Daniele Bonadiman, Yi-An Lai, Arshit Gupta, Dan Roth, Katrin Kirchhoff, Saab Mansour, James Yipeng Huang
Life of this patent
Estimating the term…
Abstract
The application is directed to receiving an alignment model and a prompt to evaluate and generate a response using a large language model. The large language model populates tokens for potential responses based on framework parameters and evaluates the potential responses using an alignment model. The alignment model can be utilized or selected for a prompt at decoding-time and can be used to select a potential response to use to create an output.
Claims19
A method comprising:
- as implemented by a hardware processor configured to execute computer-executable instructions:
- receiving, via user interaction with a reward model selection interface, a user-selected alignment model;
- receiving, at a large language model, a prompt, wherein the large language model is trained using a first set of constraints;
- populating one or more tokens using the large language model based on the prompt;
- identifying one or more framework parameters for populating additional tokens;
- based on the one or more framework parameters:
- creating a first beam and a second beam;
- and
- populating the first beam with a first subset of the additional tokens and the second beam with a second subset of the additional tokens;
- evaluating the first beam and the second beam based on a reward model, wherein the reward model is the user-selected alignment model trained using a second set of constraints;
- calculating a first preference score for the first beam and a second preference score for the second beam using the reward model;
- determining, based on the first preference score for the first beam, that the first beam does not satisfy a threshold associated with the reward model;
- selecting the second beam, wherein the second beam satisfies the threshold associated with the reward model;
- creating a response based on the second beam;
- and
- outputting the response.
A system comprising:
- a data storage medium to store computer-executable instructions;
- and
- a processor in communication with the data storage medium, wherein the processor executes the computer-executable instructions to at least:
- receive, at a large language model, a prompt;
- populate one or more tokens using the large language model based on the prompt;
- identify one or more framework parameters for populating additional tokens;
- based on the one or more framework parameters:
- create a first beam and a second beam;
- and
- populate the first beam with a first subset of the additional tokens and the second beam with a second subset of the additional tokens;
- evaluate the first beam and the second beam based on a reward model, wherein the reward model is a user-selected alignment model selected via user interaction with a reward model selection interface;
- calculate a first preference score for the first beam and a second preference score for the second beam using the reward model;
- determine, based on the first preference score for the first beam, that the first beam does not meet a threshold associated with the reward model;
- select the second beam, wherein the second beam satisfies the threshold associated with the reward model;
- and
- transmit the second beam.
A non-transitory computer-readable medium storing computer-executable instructions that, when executed by a computing system within a cloud provider network, cause the computing system to perform operations comprising:
- receive, at an artificial intelligence (AI) model, a prompt;
- populate one or more tokens using the AI model;
- identify one or more framework parameters for populating additional tokens;
- based on the one or more framework parameters:
- create a first beam and a second beam;
- and
- populate the first beam with a first subset of the additional tokens and the second beam with a second subset of the additional tokens;
- evaluate the first beam and the second beam based on a reward model, wherein the reward model comprises an alignment model trained to follow one or more constitutional principles;
- calculate, based on the alignment model, a first alignment score for the first beam and a second alignment score for the second beam;
- and
- determine, based at least in part on the first alignment score and the second alignment score, that the second beam better satisfies the reward model than the first beam.
Dependencies are inferred from the claim text.