# US 12,602,270 B1 · KV-cache streaming for improved performance and fault tolerance in generative model serving

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,270 B1
- Document ID: US12602270B1
- Status: Granted
- Kind: B1
- Application number: 18/476,422
- Filed: 2023-09-28
- Granted: 2026-04-14
- Earliest claimed: 2023-09-28
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Amar Phanishayee; Jakub Michał Tarnawski; Foteini Strati; Sara Jane McAllister
- Web page: https://patentgrep.com/patent/US12602270B1

## CPC

- G06F 9/544: {Buffers; Shared memory; Pipes}
- G06N 3/0475: Generative networks
- G06F 9/4806: {Task transfer initiation or dispatching}
- G06F 12/0862: with prefetch
- G06N 3/045: Combinations of networks
- G06N 3/063: using electronic means
- G06F 2212/602: Details relating to cache prefetching
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method of serving a generative transformer model includes determining a batch size to use in processing inference requests and allocating at least one prompt pipeline and at least on token pipeline to the generative transformer model to process the batch of inference requests. The number of prompt pipelines and the number of token pipelines, and the depths of the pipelines are determined based on the batch size, an average prompt length, a cache requirement per stage, and a memory footprint of model weights for the generative model per stage using a resource allocator component of the model serving system. Cache streaming is used to stream prompt cache from prompt pipelines to token pipelines to generate tokens. Cache streaming involves gather-copy operations which may be performed using compute kernels.

## Claims

### Claim 1

A model serving system comprising:
a processor; and
a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor alone or in combination with other processors, cause the model serving system to perform functions of:
receiving a plurality of inference requests for a generative transformer model;
determining a batch size to use in processing the plurality of inference requests;
allocating a prompt pipeline and a token pipeline to the generative transformer model to process the plurality of inference requests;
dispatching a batch of inference requests of the batch size to the prompt pipeline;
processing a first prompt from the batch of inference requests through the prompt pipeline to generate a prompt output for the first prompt;
storing the prompt output in a first compute cache;
streaming the prompt output from the first compute cache to a first stage of the token pipeline;
processing the prompt output through the token pipeline to generate a token output for the first prompt; and
outputting the token output to a result processing component of the model serving system.

### Claim 2 (depends on 1)

The model serving system of claim 1, wherein the functions further comprise:
storing the token output in a second compute cache; and
streaming the token output from the second compute cache to a remote memory or a persistent storage.

### Claim 3 (depends on 2)

The model serving system of claim 2, wherein the functions further comprise:
in response to detecting a fault or failure causing the token output in the second compute cache to be cleared and/or corrupted, streaming the token output from the remote memory or the persistent storage to the second compute cache.

### Claim 4 (depends on 2)

The model serving system of claim 2, wherein the token output includes the prompt output.

### Claim 5 (depends on 4)

The model serving system of claim 4, wherein the streaming of the second compute cache further comprises:
performing a gather operation to gather the prompt output and the token output from the second compute cache into a tensor, and
performing a copy operation to copy the tensor to the remote memory or the persistent storage.

### Claim 6 (depends on 5)

The model serving system of claim 5, wherein the gather operation and the copy operation are performed using compute kernels.

### Claim 7 (depends on 2)

The model serving system of claim 2, wherein:
the first compute cache and the second compute cache each comprise a Key-Value Cache, the Key-Value Cache including a Key Cache and a Value Cache, the Key cache being used to store output of token pipelines, and the Value cache being used to store output of prompt pipelines.

### Claim 8 (depends on 1)

The model serving system of claim 1, further comprising:
determining a number of prompt pipelines, a number of stages per prompt pipeline, a number of token pipelines, and a number of stages per token pipeline to utilize for the generative transformer model based on the batch size, an average prompt length, a cache requirement per stage, and a memory footprint of model weights for the generative transformer model per stage using a resource allocator component of the model serving system.

### Claim 9 (depends on 8)

The model serving system of claim 8, wherein:
the generative transformer model includes a plurality of transformer layers, and
determining the number of prompt pipelines, the number of stages per prompt pipeline, the number of token pipelines, and the number of stages per token pipeline includes determining a number of transformer layers per stage for the prompt pipelines and the token pipelines.

### Claim 10

A method of serving a generative transformer model, the method comprising:
receiving a plurality of inference requests for the generative transformer model;
processing the plurality of inference requests using a prompt pipeline to generate compute values, each of the compute values being representative of a prompt of an inference request:
storing the compute values for the plurality of inference requests in an intermediate memory for a first pipeline stage, the first pipeline stage corresponding to a token pipeline that is separate from the prompt pipeline;
streaming the compute values for a first batch of inference requests from the intermediate memory to a cache of the first pipeline stage;
processing the compute values for the first batch of inference requests at the first pipeline stage to generate new compute values for the first batch of inference requests;
streaming the new compute values for the first batch of inference requests from the cache to the intermediate memory;
streaming the compute values for a second batch of inference requests from the intermediate memory to the cache;
processing the compute values for the second batch of inference requests at the first pipeline stage to generate new compute values for the second batch of inference requests; and streaming the new compute values for the second batch of inference requests from the cache to the intermediate memory.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
streaming the compute values for a third batch of inference requests from the intermediate memory to the cache; and
processing the compute values for the third batch of inference requests at the first pipeline stage to generate new compute values for the third batch of inference requests; and
streaming the new compute values for the third batch of inference requests from the cache to the intermediate memory.

### Claim 12 (depends on 11)

The method of claim 11, wherein:
the cache includes a compute cache and a prefetch cache, and
prior to processing the compute values for the first batch of inference requests, the compute values for the first batch of inference requests are streamed from the intermediate memory to the compute cache and the compute values for the second batch of inference requests are streamed from the intermediate memory to the prefetch cache, and
prior to processing the compute values for the second batch of inference requests, the compute values for the second batch of inference requests are copied from the prefetch cache to the compute cache and the compute values for the third batch of inference requests are streamed from the intermediate memory to the prefetch cache.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
after receiving the plurality of inference requests,
storing the compute values in a prompt cache; and
streaming the compute values from the prompt cache to the intermediate memory.

### Claim 14 (depends on 10)

The method of claim 10, wherein the cache comprises a Key-Value Cache.

### Claim 15

A method of serving a generative transformer model, the method comprising:
storing compute values for a first batch of inference requests in a first memory for a first pipeline, the first pipeline corresponding to at least one transformer layer of the generative transformer model;
processing the compute values for the first batch of inference requests through the first pipeline to generate new compute values for the first batch of inference requests; and
while the compute values for the first batch of inference requests are being processed at the first pipeline, streaming the compute values from the first memory to a second memory for storage,
wherein:
the first pipeline is a prompt pipeline,
the first memory is a prompt cache,
the compute values correspond to prompt inputs for the prompt pipeline, and
the second memory is for a token pipeline that is separate from the prompt pipeline.

### Claim 16 (depends on 15)

The method of claim 15, wherein:
the prompt pipeline implements a plurality of transformer layers for the generative transformer model,
each of the transformer layers processes the prompt cache to generate an output that is added to the prompt cache,
the output of each transformer layer is streamed to the second memory while a next transformer layer processes the prompt cache.

### Claim 17 (depends on 15)

The method of claim 15, wherein:
the second memory is a remote memory and/or persistent storage used for fault recovery.

### Claim 18 (depends on 17)

The method of claim 17, wherein:
a token cache is streamed to the remote memory and/or persistent storage while the token cache is being processed by the token pipeline to generate next computed values for the first batch of inference requests.