Stateless caching for code generation using generative artificial intelligence

A non-transitory computer readable storage medium storing instructions which, when executed, cause a processing device to:

• receive a sequence of input tokens, wherein the sequence comprises:
• a prefix comprising one or more input tokens occurring before a reference point in a computer program, a suffix occurring after the prefix in the sequence and comprising one or more input tokens occurring after the reference point in the computer program, and retrieved data occurring after the suffix in the sequence and comprising one or more input tokens based on information accessed from one or more data sources;
• load a portion of an embedding of a previously received sequence corresponding to a leading portion of the previously received sequence that matches a leading portion of the sequence;
• and
• generate, by a code generation model, suggested code to be inserted into the computer program at the reference point, wherein the suggested code is generated based on the portion of the embedding of the previously received sequence.

A system comprising:

• a memory;
• a processing device, operatively coupled to the memory, the processing device configured to:
• receive a sequence of input tokens, wherein the sequence comprises:
• a prefix comprising one or more input tokens occurring before a reference point in a computer program, a suffix occurring after the prefix in the sequence and comprising one or more input tokens occurring after the reference point in the computer program, and retrieved data occurring after the suffix in the sequence and comprising one or more input tokens based on information accessed from one or more data sources;
• load a portion of an embedding of a previously received sequence corresponding to a leading portion of the previously received sequence that matches a leading portion of the sequence;
• and
• generate, by a code generation model, suggested code to be inserted into the computer program at the reference point, wherein the suggested code is generated based on the portion of the embedding of the previously received sequence.