# US 2026/0064937 A1 · Text generation methods and apparatuses, storage medium devices, and program products

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0064937 A1
- Document ID: US20260064937A1
- Status: Published application
- Kind: A1
- Application number: 18/958,985
- Filed: 2024-11-25
- Published: 2026-03-05
- Earliest claimed: 2024-08-27
- Assignees: Alipay (Hangzhou) Information Technology Co., Ltd.
- Applicants: Alipay (Hangzhou) Information Technology Co., Ltd.
- Inventors: Changxu Shao; Yuhong Guo; Junping Zhao
- Web page: https://patentgrep.com/patent/US20260064937A1

## CPC

- G06F 40/126: Character encoding
- G06F 16/3344: {using natural language analysis}
- G06F 40/30: Semantic analysis
- G06F 40/56: Natural language generation
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

This specification provides text generation methods, apparatuses, and storage medium devices. One method includes the following operations. In an iteration of a plurality of iterations under a large language model (LLM): estimating a first text sequence following a current text sequence based on a speculative decoding method, forming a plurality of candidate sequences based on the current text sequence and subsequences of the first text sequence, allocating logical blocks to text units in the plurality of candidate sequences in a key-value cache, to store attention information of the text units, mapping the allocated logical blocks to physical blocks based on a first criterion, and determining, by the LLM, a newly generated text unit in the iteration by using attention information of each candidate sequence in the key-value cache, to form a current text sequence for a next iteration of the plurality of iterations.

## Claims

### Claim 1

A method for generating an output text for an input text based on performing a plurality of iterations, the method comprising:
in an iteration of the plurality of iterations under a large language model (LLM):
estimating a first text sequence following a current text sequence based on a speculative decoding method;
forming a plurality of candidate sequences based on the current text sequence and subsequences of the first text sequence;
allocating logical blocks to text units in the plurality of candidate sequences in a key-value cache, to store attention information of the text units;
mapping the allocated logical blocks to physical blocks based on a first criterion, wherein the first criterion comprises a plurality of logical blocks allocated to a same text unit in the plurality of candidate sequences are mapped to a same physical block; and
determining, by the LLM, a newly generated text unit in the iteration by using attention information of each candidate sequence in the key-value cache, to form a current text sequence for a next iteration of the plurality of iterations.

### Claim 2 (depends on 1)

The method according to claim 1, wherein logical storage spaces corresponding to the key-value cache forms a matrix, and a logical block in a row in the matrix stores a text unit in a candidate sequence; and
the allocating logical blocks to text units in the plurality of candidate sequences comprises:
adding a new row to the matrix for each candidate sequence; and
adding a logical block to the row for a text unit in the candidate sequence.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the estimating a first text sequence following a current text sequence based on a speculative decoding method comprises:
querying a vocab table by using the current text sequence, to obtain the matching first text sequence, wherein the vocab table comprises a plurality of mapping relationships, each of the mapping relationships is between a text sequence and a continuation text sequence following the text sequence, and the vocab table is obtained based on a text generated by the LLM.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the vocab table comprises a global vocab table and a local vocab table, the global vocab table records a mapping relationship in which a co-occurrence frequency of sequences in the text generated by the LLM is greater than a first frequency threshold, the local vocab table records a mapping relationship in which a co-occurrence frequency of sequences in a text generated for the input text is greater than a second frequency threshold, and the local vocab table is added to the global vocab table after the output text is generated.

### Claim 5 (depends on 3)

The method according to claim 3, wherein the mapping relationship in the vocab table corresponds to a score value determined based on a co-occurrence frequency; and
the querying a vocab table by using the current text sequence, to obtain the matching first text sequence comprises:
performing query and matching for the current text sequence in the vocab table; and
when the vocab table comprises a plurality of matching text sequences, determining a text sequence with a highest score value as the first text sequence.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the determining a newly generated text unit in the iteration comprises:
predicting, by the LLM, a next text unit for each candidate sequence by using the attention information of each candidate sequence in the key-value cache;
sequentially forming a second text sequence for next text units respectively predicted for the plurality of candidate sequences;
comparing the second text sequence with the first text sequence; and
determining the newly generated text unit based on a comparison result.

### Claim 7 (depends on 6)

The method according to claim 6, further comprising:
determining a first incorrect text unit based on the comparison result, wherein the first incorrect text unit exists in the first text sequence and does not exist at a corresponding location in the second text sequence;
determining a text unit following the first incorrect text unit in the first text sequence as a second incorrect text unit; and
for each incorrect text unit in the first incorrect text unit and the second incorrect text unit, returning a logical block and a physical block corresponding to attention information of the incorrect text unit in the key-value cache.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the input text comprises a plurality of user request texts, the current text sequence comprises a plurality of sequence segments, a single sequence segment corresponds to an output text generated for a single user request text, and the first text sequence comprises a plurality of continuation sequences respectively estimated for the plurality of sequence segments.

### Claim 9 (depends on 8)

The method according to claim 8, wherein
before the estimating a first text sequence following a current text sequence based on a speculative decoding method, the method further comprises:
determining a hardware computing power indicator and a first quantity of text units in a target request text, wherein the hardware computing power indicator indicates a maximum quantity of text units whose attention information can be simultaneously computed, and the target request text is a user request text for which a first text unit in an output text is not generated when the iteration starts;
determining an available computing power indicator based on a difference between the maximum quantity of text units and the first quantity; and
determining a length of the first text sequence based on the available computing power indicator.

### Claim 10 (depends on 8)

The method according to claim 8, wherein the estimating a first text sequence following a current text sequence based on a speculative decoding method comprises:
performing the speculative decoding when the LLM generates at least one output text unit for each of the plurality of user request texts, wherein the current text sequence comprises a plurality of sequence segments corresponding to the plurality of user request texts.

### Claim 11

An apparatus for generating an output text for an input text based on performing a plurality of iterations, the apparatus comprising:
at least one processor; and
one or more memories coupled to the at least one processor and storing programming instructions for execution by the at least one processor to perform operations comprising:
in an iteration of the plurality of iterations under a large language model (LLM):
estimating a first text sequence following a current text sequence based on a speculative decoding method;
forming a plurality of candidate sequences based on the current text sequence and subsequences of the first text sequence;
allocating logical blocks to text units in the plurality of candidate sequences in a key-value cache, to store attention information of the text units;
mapping the allocated logical blocks to physical blocks based on a first criterion, wherein the first criterion comprises a plurality of logical blocks allocated to a same text unit in the plurality of candidate sequences are mapped to a same physical block; and
determining, by the LLM, a newly generated text unit in the iteration by using attention information of each candidate sequence in the key-value cache, to form a current text sequence for a next iteration of the plurality of iterations.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein logical storage spaces corresponding to the key-value cache forms a matrix, and a logical block in a row in the matrix stores a text unit in a candidate sequence; and
the allocating logical blocks to text units in the plurality of candidate sequences comprises:
adding a new row to the matrix for each candidate sequence; and
adding a logical block to the row for a text unit in the candidate sequence.

### Claim 13 (depends on 11)

The apparatus according to claim 11, wherein the estimating a first text sequence following a current text sequence based on a speculative decoding method comprises:
querying a vocab table by using the current text sequence, to obtain the matching first text sequence, wherein the vocab table comprises a plurality of mapping relationships, each of the mapping relationships is between a text sequence and a continuation text sequence following the text sequence, and the vocab table is obtained based on a text generated by the LLM.

### Claim 14 (depends on 13)

The apparatus according to claim 13, wherein the vocab table comprises a global vocab table and a local vocab table, the global vocab table records a mapping relationship in which a co-occurrence frequency of sequences in the text generated by the LLM is greater than a first frequency threshold, the local vocab table records a mapping relationship in which a co-occurrence frequency of sequences in a text generated for the input text is greater than a second frequency threshold, and the local vocab table is added to the global vocab table after the output text is generated.

### Claim 15 (depends on 13)

The apparatus according to claim 13, wherein the mapping relationship in the vocab table corresponds to a score value determined based on a co-occurrence frequency; and
the querying a vocab table by using the current text sequence, to obtain the matching first text sequence comprises:
performing query and matching for the current text sequence in the vocab table; and
when the vocab table comprises a plurality of matching text sequences, determining a text sequence with a highest score value as the first text sequence.

### Claim 16 (depends on 11)

The apparatus according to claim 11, wherein the determining a newly generated text unit in the iteration comprises:
predicting, by the LLM, a next text unit for each candidate sequence by using the attention information of each candidate sequence in the key-value cache;
sequentially forming a second text sequence for next text units respectively predicted for the plurality of candidate sequences;
comparing the second text sequence with the first text sequence; and
determining the newly generated text unit based on a comparison result.

### Claim 17 (depends on 16)

The apparatus according to claim 16, the operations further comprising:
determining a first incorrect text unit based on the comparison result, wherein the first incorrect text unit exists in the first text sequence and does not exist at a corresponding location in the second text sequence;
determining a text unit following the first incorrect text unit in the first text sequence as a second incorrect text unit; and
for each incorrect text unit in the first incorrect text unit and the second incorrect text unit, returning a logical block and a physical block corresponding to attention information of the incorrect text unit in the key-value cache.

### Claim 18 (depends on 11)

The apparatus according to claim 11, wherein the input text comprises a plurality of user request texts, the current text sequence comprises a plurality of sequence segments, a single sequence segment corresponds to an output text generated for a single user request text, and the first text sequence comprises a plurality of continuation sequences respectively estimated for the plurality of sequence segments.

### Claim 19 (depends on 18)

The apparatus according to claim 18, wherein
before the estimating a first text sequence following a current text sequence based on a speculative decoding method, the method further comprises:
determining a hardware computing power indicator and a first quantity of text units in a target request text, wherein the hardware computing power indicator indicates a maximum quantity of text units whose attention information can be simultaneously computed, and the target request text is a user request text for which a first text unit in an output text is not generated when the iteration starts;
determining an available computing power indicator based on a difference between the maximum quantity of text units and the first quantity; and
determining a length of the first text sequence based on the available computing power indicator.

### Claim 20

A non-transitory, computer-readable medium storing one or more instructions executable by at least one processor to perform operations comprising:
in an iteration of a plurality of iterations under a large language model (LLM):
estimating a first text sequence following a current text sequence based on a speculative decoding method;
forming a plurality of candidate sequences based on the current text sequence and subsequences of the first text sequence;
allocating logical blocks to text units in the plurality of candidate sequences in a key-value cache, to store attention information of the text units;
mapping the allocated logical blocks to physical blocks based on a first criterion, wherein the first criterion comprises a plurality of logical blocks allocated to a same text unit in the plurality of candidate sequences are mapped to a same physical block; and
determining, by the LLM, a newly generated text unit in the iteration by using attention information of each candidate sequence in the key-value cache, to form a current text sequence for a next iteration of the plurality of iterations.
