Template guided speculative decoding

A method for accelerated machine learning based sequence generation, comprising:

• generating a first portion of a sequence based on an input query using a speculative decoding process involving a draft machine learning model and a target machine learning model that has a larger number of parameters than the draft machine learning model;
• determining, by a decision mechanism operating on the input query and the first portion, that a next portion of the sequence is predicted to correspond to a template, wherein the decision mechanism comprises a binary classification model configured to generate an output for use in selecting between a template-based generative process and the draft machine learning model, and wherein the binary classification model has been trained using training data generated through an automated labeling pipeline that, for corpus contexts, generates template continuations and draft continuations, compares log-probabilities of the template continuations and the draft continuations generated using the target machine learning model, and generates labels for the corpus contexts based on the comparing of the log-probabilities;
• based on the determining that the next portion is predicted to correspond to the template, generating the next portion of the sequence, without using the draft machine learning model, through the template-based generative process involving one or more rules or a generative machine learning model having fewer parameters than the draft machine learning model;
• verifying the next portion using the target machine learning model;
• and
• appending the next portion to the sequence based on the verifying.

A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to:

• generate a first portion of a sequence based on an input query using a speculative decoding process involving a draft machine learning model and a target machine learning model that has a larger number of parameters than the draft machine learning model;
• determine, by a decision mechanism operating on the input query and the first portion, that a next portion of the sequence is predicted to correspond to a template;
• based on the determining that the next portion is predicted to correspond to the template, generate the next portion of the sequence, without using the draft machine learning model, through a template-based generative process involving one or more rules or a generative machine learning model having fewer parameters than the draft machine learning model, wherein the template-based generative process produces a multi-token segment in a single operation;
• verify the next portion using the target machine learning model, wherein the verifying of the next portion using the target machine learning model comprises a single-shot acceptance determination for the multi-token segment;
• and
• append the next portion to the sequence based on the verifying.