Speculative decoding in autoregressive generative artificial intelligence models

A processor-implemented method, comprising:

• generating, based on an input prompt and a generative artificial intelligence model, a first plurality of sets of tokens, each set of tokens in the first plurality of sets of tokens corresponding to a first portion of a candidate response to the input prompt;
• generating, using the generative artificial intelligence model, a second plurality of sets of tokens, each set of tokens in the second plurality of sets of tokens corresponding to a second portion of the candidate response to the input prompt based on the first plurality of sets of tokens;
• while generating the second plurality of sets of tokens, selecting a set of tokens from the first plurality of sets of tokens;
• and
• outputting the selected set of tokens from the first plurality of tokens and an associated set of tokens from the second plurality of tokens as a response to the input prompt.

A processing system comprising:

• means for generating, based on an input prompt and a generative artificial intelligence model, a first plurality of sets of tokens, each set of tokens in the first plurality of sets of tokens corresponding to a first portion of a candidate response to the input prompt;
• means for generating, using the generative artificial intelligence model, a second plurality of sets of tokens, each set of tokens in the second plurality of sets of tokens corresponding to a second portion of the candidate response to the input prompt based on the first plurality of sets of tokens;
• means for selecting, while generating the second plurality of sets of tokens, a set of tokens from the first plurality of sets of tokens;
• and
• means for outputting the selected set of tokens from the first plurality of tokens and an associated set of tokens from the second plurality of tokens as a response to the input prompt.

A non-transitory computer-readable medium having executable instructions stored thereon which, when executed by one or more processors, perform an operation comprising:

• generating, based on an input prompt and a generative artificial intelligence model, a first plurality of sets of tokens, each set of tokens in the first plurality of sets of tokens corresponding to a first portion of a candidate response to the input prompt;
• generating, using the generative artificial intelligence model, a second plurality of sets of tokens, each set of tokens in the second plurality of sets of tokens corresponding to a second portion of the candidate response to the input prompt based on the first plurality of sets of tokens;
• while generating the second plurality of sets of tokens, selecting a set of tokens from the first plurality of sets of tokens;
• and
• outputting the selected set of tokens from the first plurality of tokens and an associated set of tokens from the second plurality of tokens as a response to the input prompt.