Large language model proxy aggregator
AssigneeSap SE
InventorJasmond Ming Quan Loh
Life of this patent
Estimating the term…
Abstract
In an example embodiment, a filtering system is introduced to optimize the organization and processing of LLM request. By carefully categorizing queries based on their input prompt lengths and anticipated output token numbers, the workflow is streamlined prior to submission to an LLM inference server. Each bucket corresponds to a different range of number of tokens (both input and output combined). Each bucket also has a size, indicating the maximum number of requests that can be placed in the bucket. Each request is assigned to a bucket when it is received, and when a bucket is filled the requests in the bucket are batched together and send to the LLM for processing.
Claims20
A system comprising:
- at least one hardware processor;
- a non-transitory computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
- receiving a first large language model (LLM) prompt;
- determining a number of input tokens in the first LLM prompt;
- placing the first LLM prompt in a first bucket of a plurality of buckets in a bucket cache, based on the number of input tokens;
- upon detecting that the first bucket is full, batching all LLM prompts in the first bucket into a single batch;
- sending the first batch to an LLM for processing;
- receiving, from the LLM, a batch of LLM results;
- identifying an LLM result, in the batch of LLM results, corresponding to the first LLM prompt;
- and
- returning the LLM result corresponding to the first LLM prompt as a response to the first LLM prompt.
A method comprising:
- receiving a first large language model (LLM) prompt;
- determining a number of input tokens in the first LLM prompt;
- placing the first LLM prompt in a first bucket of a plurality of buckets in a bucket cache, based on the number of input tokens;
- upon detecting that the first bucket is full, batching all LLM prompts in the first bucket into a single batch;
- sending the first batch to an LLM for processing;
- receiving, from the LLM, a batch of LLM results;
- identifying an LLM result, in the batch of LLM results, corresponding to the first LLM prompt;
- and
- returning the LLM result corresponding to the first LLM prompt as a response to the first LLM prompt.
A non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- receiving a first large language model (LLM) prompt;
- determining a number of input tokens in the first LLM prompt;
- placing the first LLM prompt in a first bucket of a plurality of buckets in a bucket cache, based on the number of input tokens;
- upon detecting that the first bucket is full, batching all LLM prompts in the first bucket into a single batch;
- sending the first batch to an LLM for processing;
- receiving, from the LLM, a batch of LLM results;
- identifying an LLM result, in the batch of LLM results, corresponding to the first LLM prompt;
- and
- returning the LLM result corresponding to the first LLM prompt as a response to the first LLM prompt.
Dependencies are inferred from the claim text.