Techniques for adaptive large language model usage

An apparatus for adaptive large language model (LLM) usage, comprising:

• one or more memories storing code;
• and
• one or more processors coupled with the one or more memories and individually or collectively operable to execute the code to cause the apparatus to:
• obtain, by a data management system (DMS), data associated with a communication session between a user and a communication service of the DMS;
• generate, by the DMS, a prompt comprising some or all of the data associated with the communication session between the user and the communication service of the DMS;
• select, from a plurality of LLMs that are available to the communication service of the DMS, an LLM based at least in part on a token size of the prompt and a token window size of the LLM and based at least in part on a response latency associated with the selected LLM;
• transmit, to the selected LLM, the prompt comprising some or all of the data associated with the communication session between the user and the communication service of the DMS;
• receive, from the selected LLM and in accordance with the response latency, a response to the prompt comprising some or all of the data associated with the communication session between the user and the communication service;
• and
• include data associated with the response from the selected LLM in the communication session between the user and the communication service of the DMS.

A non-transitory computer-readable medium storing code for adaptive large language model (LLM) usage, the code comprising instructions executable by one or more processors to:

• obtain, by a data management system (DMS), data associated with a communication session between a user and a communication service of the DMS;
• generate, by the DMS, a prompt comprising some or all of the data associated with the communication session between the user and the communication service of the DMS;
• select, from a plurality of LLMs that are available to the communication service of the DMS, an LLM based at least in part on a token size of the prompt and a token window size of the LLM and based at least in part on a response latency associated with the selected LLM;
• transmit, to the selected LLM, the prompt comprising some or all of the data associated with the communication session between the user and the communication service of the DMS;
• receive, from the selected LLM and in accordance with the response latency, a response to the prompt comprising some or all of the data associated with the communication session between the user and the communication service;
• and
• include data associated with the response from the selected LLM in the communication session between the user and the communication service of the DMS.