Prompt routing system and method
AssigneeMartian Learning, Inc.
InventorsShriyash K. Upadhyay, Etan J. Ginsberg, Dory Zidon
Life of this patent
Estimating the term…
Abstract
In variants, the method can include determining training data, determining a router, and using the router. In variants, using the router can include receiving a runtime prompt, predicting performance scores for the runtime prompt for each of a set of candidate models, optionally predicting operational metrics for responding to the runtime prompt for each of the set of candidate models, selecting a candidate model based on the predicted performance scores and optionally the predicted operational metrics, and optionally determining a response based on the runtime prompt.
Claims20
A method for routing machine learning model prompts, comprising:
- receiving a prompt;
- determining a candidate model response score for each of a set of candidate models based on the prompt, using a scoring model;
- selecting a runtime model from the set of candidate models based on the set of candidate model response scores;
- and
- facilitating runtime model response determination for the prompt using the runtime model.
A method for model routing, comprising:
- determining a routing model trained to predict a response score given a prompt, wherein the routing model is trained based on:
- a set of prompts within a set of training prompt-response pairs, each training response determined using a candidate model within a set of candidate models;
- and
- a set of training response scores, each training response score evaluating a training prompt-response pair from the set of training prompt-response pairs;
- receiving a new prompt from a user;
- determining a response score for each candidate model within the set of candidate models based on the new prompt using the routing model;
- selecting a runtime model from the set of candidate models for the new prompt based on the response scores;
- and
- facilitating determination of a response to the new prompt using the selected runtime model, wherein the response is returned to the user.
Dependencies are inferred from the claim text.