Prefetching portions of large language models

A system comprising:

• a first memory;
• a second memory;
• and
• a processor coupled to the first memory and the second memory, the processor configured to:
• receive a current token and at least one previous token generated based on a user input to a generative large language model;
• identify subnetworks within the generative large language model by processing the current token and the at least one previous token using a machine learning model;
• prefetch the subnetworks from the first memory into the second memory;
• and
• cause the generative large language model to generate an output based on the user input using the subnetworks in the second memory.

A non-transitory computer-readable storage medium storing instructions that, responsive to execution by a processor, cause the processor to perform operations comprising:

• receiving a token computed based on a user input to a generative large language model;
• identifying portions of layers of the generative large language model by processing the token using a machine learning model;
• writing the portions of the layers into a memory;
• and
• generating an output based on the user input and the token with the generative large language model using the portions of the layers in the memory.