Speculative Decoding Systems and Methods Having Improved Throughput

A method for speculative decoding, comprising:

• receiving a plurality of requests from one or more users;
• assigning at least one draft model to each request of the plurality of requests;
• at each processing cycle of a plurality of processing cycles:
• receiving one or more draft tokens from the at least one draft model;
• and
• verifying, by a target model, the one or more draft tokens from the at least one draft model;
• and
• generating a plurality of responses to the plurality of requests using the verified one or more draft tokens;
• wherein the target model is implemented as a pipeline comprising a plurality of pipeline stages, each pipeline stage of the plurality of pipeline stages configured to receive a plurality of batches, each batch of the plurality of batches accommodating a respective plurality of tokens;
• wherein a first amount of the plurality of pipeline stages, a second amount of the plurality of batches, and a third amount of the plurality of tokens is selected to optimize throughput of the system for the one or more users.

One or more non-transitory, computer-readable media storing instructions that, when implemented, cause one or more processors to perform operations, the operations comprising:

• receiving a plurality of requests from one or more users;
• assigning at least one draft model to each request of the plurality of requests;
• at each processing cycle of a plurality of processing cycles:
• receiving one or more draft tokens from the at least one draft model;
• and
• verifying, by a target model, the one or more draft tokens from the at least one draft model;
• and
• generating a plurality of responses to the plurality of requests using the verified one or more draft tokens;
• wherein the target model is implemented as a pipeline comprising a plurality of pipeline stages, each pipeline stage of the plurality of pipeline stages configured to receive a plurality of batches, each batch of the plurality of batches accommodating a respective plurality of tokens;
• wherein a first amount of the plurality of pipeline stages, a second amount of the plurality of batches, and a third amount of the plurality of tokens is selected to optimize throughput of the system for the one or more users.