US 12,602,270 B1Granted

KV-cache streaming for improved performance and fault tolerance in generative model serving

AssigneeMicrosoft Technology Licensing, LLC

InventorsAmar Phanishayee, Jakub Michał Tarnawski, Foteini Strati, Sara Jane McAllister

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method of serving a generative transformer model includes determining a batch size to use in processing inference requests and allocating at least one prompt pipeline and at least on token pipeline to the generative transformer model to process the batch of inference requests. The number of prompt pipelines and the number of token pipelines, and the depths of the pipelines are determined based on the batch size, an average prompt length, a cache requirement per stage, and a memory footprint of model weights for the generative model per stage using a resource allocator component of the model serving system. Cache streaming is used to stream prompt cache from prompt pipelines to token pipelines to generate tokens. Cache streaming involves gather-copy operations which may be performed using compute kernels.

Claims18

Dependent claims

A model serving system comprising:

a processor;
and
a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor alone or in combination with other processors, cause the model serving system to perform functions of:
receiving a plurality of inference requests for a generative transformer model;
determining a batch size to use in processing the plurality of inference requests;
allocating a prompt pipeline and a token pipeline to the generative transformer model to process the plurality of inference requests;
dispatching a batch of inference requests of the batch size to the prompt pipeline;
processing a first prompt from the batch of inference requests through the prompt pipeline to generate a prompt output for the first prompt;
storing the prompt output in a first compute cache;
streaming the prompt output from the first compute cache to a first stage of the token pipeline;
processing the prompt output through the token pipeline to generate a token output for the first prompt;
and
outputting the token output to a result processing component of the model serving system.

10 more elements

A method of serving a generative transformer model, the method comprising:

receiving a plurality of inference requests for the generative transformer model;
processing the plurality of inference requests using a prompt pipeline to generate compute values, each of the compute values being representative of a prompt of an inference request:
storing the compute values for the plurality of inference requests in an intermediate memory for a first pipeline stage, the first pipeline stage corresponding to a token pipeline that is separate from the prompt pipeline;
streaming the compute values for a first batch of inference requests from the intermediate memory to a cache of the first pipeline stage;
processing the compute values for the first batch of inference requests at the first pipeline stage to generate new compute values for the first batch of inference requests;
streaming the new compute values for the first batch of inference requests from the cache to the intermediate memory;
streaming the compute values for a second batch of inference requests from the intermediate memory to the cache;
processing the compute values for the second batch of inference requests at the first pipeline stage to generate new compute values for the second batch of inference requests;
and streaming the new compute values for the second batch of inference requests from the cache to the intermediate memory.

6 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/476,422Filed 28 Sep 2023
Granted: 14 Apr 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification9 codes

G06F 9/544
Interprogram communication: buffers; Shared memory; Pipes
G06N 3/0475
Architecture, e.g. interconnection topology: generative networks
G06F 9/4806
Program initiating; Program switching, e.g. by interrupt: task transfer initiation or dispatching
G06F 12/0862
Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches: with prefetch

Family

No related US applications are recorded in the indexed filing history.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key