Generative model evaluation and improvement pipeline
AssigneeDropbox, Inc.
InventorsAndrew Donald Yates, Daniel Bosnic Hill, Hans Sayyadi
Life of this patent
Estimating the term…
Abstract
This disclosure describes systems that evaluating generative queries and modifying generative models (e.g., large language models or LLMs) and/or their system prompts based on the evaluation. The disclosed systems can generate and maintain a generative model functionality codex that stores tags for prompts and responses (to capture behaviors and intents of users and LLMs or other generative models). Using the codes, the disclosed systems can update evaluation model generation and can update queries and/or model parameters based on the evaluation. The disclosed systems can further distill models using updated queries and/or learned model parameters.
Claims20
A computer-implemented method comprising:
- receiving a query from a client device interacting with a generative model;
- determining an evaluation tag corresponding to the query by comparing the query with parameters of evaluation tags stored in a generative model functionality codex;
- generating a modified system prompt for the generative model by combining an initial system prompt of the generative model with parameters of the evaluation tag corresponding to the query;
- generating, using the generative model to process the query and the modified system prompt, an evaluation data object indicating a measure of success by the generative model in generating an output responding to the query according to the evaluation tag;
- generating an explanation by using the generative model to determine an alignment between the output and an expected output associated with the evaluation tag;
- and
- providing, in response to using the generative model to determine the alignment, the explanation for display at the client device.
A system comprising:
- at least one processor;
- and
- at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:
- receive a query from a client device interacting with a generative model;
- determine one or more evaluation tags corresponding to the query by comparing the query with parameters of evaluation tags stored in a generative model functionality codex;
- generate a modified system prompt for the generative model by combining an initial system prompt of the generative model with parameters of the one or more evaluation tags corresponding to the query;
- generate, using the generative model to process the query and the modified system prompt, an evaluation data object indicating a measure of success by the generative model in generating an output responding to the query according to the one or more evaluation tags;
- generate an explanation by using the generative model to determine an alignment between the output and an expected output associated with the one or more evaluation tags;
- and
- provide, in response to using the generative model to determine the alignment, the explanation for display at the client device.
A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:
- receive a query from a client device interacting with a generative model;
- determine an evaluation tag corresponding to the query by comparing the query with evaluation tags stored in a generative model functionality codex;
- generate a modified system prompt for the generative model by combining an initial system prompt of the generative model with parameters of the evaluation tag corresponding to the query;
- generate, using the generative model, an evaluation data object indicating a measure of success by the generative model in generating an output responding to the query according to the evaluation tag;
- generate an explanation by using the generative model to determine an alignment between the output and an expected output associated with the evaluation tag;
- and
- provide, in response to using the generative model to determine the alignment, the explanation for display at the client device.
Dependencies are inferred from the claim text.