Expert selection from mixture of experts in large language models

A method comprising:

• selecting, in a global mode of a selection scheme, a global expert set from a mixture of experts (MoE) to generate a selected global expert set for each layer for an entire layer set prior to an inference phase in a machine learning (ML) model;
• and
• pre-fetching, in the global mode, the selected global expert set from a first memory into a second memory,
• wherein the selected global expert set includes one or more global hot experts.

A system comprising:

• an input token generator configured to generate an input token set;
• a layer set configured to process the input token set;
• an expert selector comprising:
• a global selector configured to manage a selection scheme having at least one of a global mode or a local mode, wherein in the global mode the global selector selects a global expert set from a mixture of experts (MoE) to generate a selected global expert set for each layer for an entire layer set prior to an inference phase in a machine learning (ML) model, the global selector generating a selected global expert set for the each layer for the entire layer set;
• and
• a pre-fetcher configured to pre-fetch in the global mode the selected global expert set from a first memory into a second memory,
• wherein the selected global expert set includes one or more global hot experts.