All-shot training of large language models

An apparatus for performing in-context training for a machine learning (ML) model, the apparatus comprising:

• one or more processors;
• a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the apparatus to;
• receive an ML model comprising a context window of a predetermined size;
• receive a set of in-context prompt/completion pairs prepared for a target task;
• construct a set of token sequences based on the set of in-context prompt/completion pairs, wherein each token sequence in the set of token sequences can be fitted into the context window;
• and
• perform a sequence of in-context training passes using the set of token sequences to train the ML model to generate a next token in accordance with the target task.

A system for performing in-context training for a machine learning (ML) model, the system comprising:

• one or more processors;
• a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the system to:
• receive an ML model comprising a context window of a predetermined size;
• receive a set of in-context prompt/completion pairs prepared for a target task;
• construct a first token sequence based on the set of in-context prompt/completion pairs;
• fit the first token sequence into the context window;
• and
• perform a first in-context training pass using the first token sequence to train the ML model to generate a next token in accordance with the target task.