Generating tokens using near-memory computing

A memory system, comprising:

• one or more memory devices;
• and
• one or more controllers configured to:
• obtain, from a host system, a first command indicating one or more input tokens associated with a large language model;
• generate, based on the one or more input tokens, one or more first tokens using one or more first parameters, the one or more first parameters having a first fidelity;
• provide, to the host system, the one or more first tokens;
• obtain, from the host system, a second command indicating one or more second tokens associated with the one or more input tokens;
• generate, based on the one or more second tokens, one or more third tokens using one or more second parameters, the one or more second parameters having a second fidelity different than the first fidelity;
• and
• provide, to the host system, the one or more third tokens.

A host system, comprising:

• one or more controllers configured to:
• provide, to a memory system, a first command indicating a prompt associated with a large language model, the first command further indicating that the memory system is to generate one or more first tokens using one or more first parameters of the large language model, the one or more first parameters having a first fidelity;
• obtain, from the memory system, the one or more first tokens;
• generate, using the one or more first tokens and one or more second parameters having a second fidelity, one or more second tokens;
• and
• provide a second command indicating the one or more second tokens to the memory system, the second command further indicating that the memory system is to generate one or more third tokens using the one or more second tokens.

A host system, comprising:

• one or more controllers configured to:
• provide, to a memory system, a first command indicating a prompt associated with a large language model, the first command further indicating that the memory system is to generate one or more first tokens using one or more first parameters of the large language model, the one or more first parameters having a first fidelity;
• obtain, from the memory system, the one or more first tokens;
• generate, using the one or more first tokens and one or more second parameters having a second fidelity, one or more second tokens;
• select a third fidelity based on a comparison of the one or more first tokens with the one or more second tokens;
• and
• provide a second command indicating the one or more second tokens to the memory system, the second command further indicating that the memory system is to generate one or more third tokens using one or more third parameters having the third fidelity.

A system, comprising;

• a host system;
• a memory apparatus;
• an interface between the host system and the memory apparatus;
• and
• one or more controllers configured to:
• communicate, via the interface and to the memory apparatus, a first command indicating a prompt associated with a large language model, the first command further indicating that the memory apparatus is to generate one or more first tokens using one or more first parameters of the large language model, the one or more first parameters having a first fidelity;
• communicate, via the interface and to the host system, the one or more first tokens;
• and
• communicate, via the interface and to the memory apparatus, a second command indicating one or more second tokens, the second command further indicating that the memory apparatus is to generate one or more third tokens using one or more second parameters of the large language model, the one or more second parameters having a second fidelity.