Data management coprocessor for the speculative inference of a large language model

A data management coprocessor comprising:

• a cache manager that allocates a cache buffer in a memory separate from the data management coprocessor;
• a large language model (LLM) weight predictor that predicts a subset of LLM weights, the subset LLM weights for the use of generating a next token by an LLM executing in an AI accelerator;
• a direct memory access (DMA) initiator that initiates caching of the subset of LLM weights into the cache buffer;
• and
• a misprediction manager that rolls back a state of the LLM executing in the AI accelerator when a confidence score for the next token meets a criterion indicating a misprediction.

A computer system, comprising:

• a processor system;
• an artificial intelligence (AI) accelerator;
• and
• a data management coprocessor that:
• allocates a cache buffer in a memory separate from the data management coprocessor;
• predicts a subset of large language model (LLM) weights, the subset LLM weights for the use of generating a next token by an LLM executing in the AI accelerator;
• initiates caching of the subset of LLM weights into the cache buffer;
• and
• rolls back a state of the LLM executing in the AI accelerator when a confidence score for the next token meets a criterion indicating a misprediction.