Large language model inferencing acceleration techniques

A processor configured to:

• generate a plurality of tokens from a prompt to a large language model (LLM);
• in one or more iterations:
• using a first neural network, output a set of speculative decoding parameters selected from a plurality of sets of speculative decoding parameters;
• and
• perform speculative decoding using the set of speculative decoding parameters to generate a subsequent set of tokens appended to the plurality of tokens based on the prompt or from a previous iteration to generate an updated plurality of tokens;
• and
• collect a runtime of the speculative decoding;
• and
• repeat the one or more iterations until the updated plurality of tokens reaches a maximum token length,
• wherein the processor is configured to train the first neural network to output sets of speculative decoding parameters to minimize a sum of runtimes of the one or more iterations.

A system comprising:

• a memory configured to store a plurality of speculative decoding parameters;
• a processor configured to, in one or more iterations:
• retrieve, from the memory, a set of speculative decoding parameters from the plurality of speculative decoding parameters;
• perform speculative decoding using the set of speculative decoding parameters to generate a subsequent set of tokens appended to a plurality of tokens generated from a prompt or from a previous iteration to generate an updated plurality of tokens;
• and
• collect a runtime of the speculative decoding;
• and
• repeat the one or more iterations until the updated plurality of tokens reaches a maximum token length,
• wherein the processor is configured to implement a first neural network to retrieve sets of speculative decoding parameters from the memory to minimize a sum of runtimes of the one or more iterations.