Adaptive caching of model responses

A system for adaptive caching of a response generated by a Large Language Model (LLM) for a received prompt, the system comprising:

• at least one memory storing machine-executable instructions;
• and
• at least one processor communicatively coupled with the at least one memory, wherein the at least one processor executes the machine-executable instructions to perform operations comprising:
• processing data associated with the response and the received prompt;
• updating a respective value of each of a plurality of features to generate a discrete time series based upon the data;
• generating a plurality of caching metrics based upon a plurality of respective values of one or more features of the plurality of features;
• generating a safety score corresponding to the plurality of caching metrics;
• and
• storing the data in a caching database, wherein
• storing the data in the caching database comprises storing a partial or no data in the caching database upon determining a size of the response of the data being greater than a threshold value, and
• the data is based at least in part upon the safety score, a response time predicted for a request associated with the received prompt, and metadata associated with the data.

A non-transitory computer-readable media comprising instructions stored thereon for adaptive caching of a response generated by a Large Language Model (LLM) for a received prompt, wherein the instructions, when executed by at least one processor of a computing system, cause the computing system to perform operations comprising:

• processing data associated with the response and the received prompt;
• updating a respective value of each of a plurality of features to generate a discrete time series based upon the data;
• generating a plurality of caching metrics based upon a plurality of respective values of one or more features of the plurality of features;
• generating a safety score corresponding to the plurality of caching metrics;
• and
• storing the data in a caching database, wherein
• storing the data in the caching database comprises storing a partial or no data in the caching database upon determining a size of the response of the data being greater than a threshold value, and
• the data is based at least in part upon the safety score, a response time predicted for a request associated with the received prompt, and metadata associated with the data.