Generative semantic caching and model distillation in natural language processing

An apparatus, comprising:

• one or more network interfaces;
• a processor coupled to the one or more network interfaces and configured to execute one or more processes;
• and
• a memory configured to store a process that is executable by the processor, the process when executed configured to:
• obtain a prompt to a query service from a client;
• determine whether a cache containing previously stored prompt-response pairs includes a match for the prompt based on a similarity metric;
• generate a response to the prompt using a student language model trained on prompt-response data from the cache;
• and
• determine, based on an assessment of the response generated by the student language model, whether to return the response generated by the student language model to the client or to escalate the prompt to a main language model for response generation.

A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:

• obtaining a prompt to a query service from a client;
• determining whether a cache containing previously stored prompt-response pairs includes a match for the prompt based on a similarity metric;
• generating a response to the prompt using a student language model trained on prompt-response data from the cache;
• and
• determining, based on an assessment of the response generated by the student language model, whether to return the response generated by the student language model to the client or to escalate the prompt to a main language model for response generation.