Speculative decoding in autoregressive generative artificial intelligence models

A processing system comprising:

• at least one memory having executable instructions stored thereon;
• and
• one or more processors configured to execute the executable instructions to cause the processing system to:
• receive a plurality of sets of tokens generated based on an input prompt and a first generative artificial intelligence model, each set of tokens in the plurality of sets of tokens comprising a sequence of tokens corresponding to a candidate response to the input prompt, wherein the plurality of sets of tokens is organized into a tree data structure and wherein a depth of the tree data structure corresponds to a parameter defining a maximum number of tokens generated by a single pass through the first generative artificial intelligence model;
• select, using a second generative artificial intelligence model and recursive adjustment of a target distribution associated with the received plurality of sets of tokens, a set of tokens from the plurality of sets of tokens;
• and
• output the selected set of tokens as a response to the input prompt.

A processor-implemented method of machine learning, comprising:

• receiving a plurality of sets of tokens generated based on an input prompt and a first generative artificial intelligence model, each set of tokens in the plurality of sets of tokens comprising a sequence of tokens corresponding to a candidate response to the input prompt, wherein:
• the plurality of sets of tokens is organized into a tree data structure;
• the tree data structure includes a plurality of levels, each level corresponding to a token in the sequence of tokens;
• and
• a number of tokens at a particular level in the tree data structure is based on a branching factor associated with an immediately prior level to the particular level in the tree data structure;
• selecting, using a second generative artificial intelligence model and recursive adjustment of a target distribution associated with the received plurality of sets of tokens, a set of tokens from the plurality of sets of tokens;
• and
• outputting the selected set of tokens as a response to the input prompt.

A processor-implemented method of machine learning, comprising:

• receiving a plurality of sets of tokens generated based on an input prompt and a first generative artificial intelligence model, each set of tokens in the plurality of sets of tokens comprising a sequence of tokens corresponding to a candidate response to the input prompt, wherein the plurality of sets of tokens is organized into a tree data structure and wherein a depth of the tree data structure corresponds to a parameter defining a maximum number of tokens generated by a single pass through the first generative artificial intelligence model;
• selecting, using a second generative artificial intelligence model and recursive adjustment of a target distribution associated with the received plurality of sets of tokens, a set of tokens from the plurality of sets of tokens;
• and
• outputting the selected set of tokens as a response to the input prompt.