# US 2025/0094709 A1 · Accelerating language model inference with dynamic multi-token sampling

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094709 A1
- Document ID: US20250094709A1
- Status: Published application
- Kind: A1
- Application number: 18/827,108
- Filed: 2024-09-06
- Published: 2025-03-20
- Earliest claimed: 2023-09-14
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Shikhar TULI; Chi-Heng Lin; Yen-Chang Hsu; Yilin Shen; Hongxia Jin
- Web page: https://patentgrep.com/patent/US20250094709A1

## CPC

- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for performing multi-token prediction by an apparatus includes receiving, from an artificial intelligence (AI) assistance device, a request for an output token sequence that is subsequent to an input token sequence indicated by the request, predicting, by a trained machine learning model, a plurality of candidate output tokens, estimating joint probability distributions of one or more combinations of the plurality of candidate output tokens, calculating joint probabilities of the one or more combinations by masking the joint probability distributions with a co-occurrence weighted mask, determining, based on the joint probabilities, whether to reduce the number of candidate output tokens included in each combination of the one or more combinations, identifying, based on the joint probabilities, a combination of the one or more combinations as the output token sequence, and outputting, to the AI assistance device, a response to the request, the response comprising the output token sequence.

## Claims

### Claim 1

A method for performing multi-token prediction by an apparatus, the method comprising:
receiving, from an artificial intelligence (AI) assistance device, a request for an output token sequence that is subsequent to an input token sequence indicated by the request;
predicting, by a trained machine learning model, a plurality of candidate output tokens, the trained machine learning model having been trained to simultaneously predict the plurality of candidate output tokens based on the input token sequence;
estimating joint probability distributions of one or more combinations of the plurality of candidate output tokens, each combination of the one or more combinations having a number of candidate output tokens that corresponds to a target language generation speed;
calculating joint probabilities of the one or more combinations by masking the joint probability distributions with a co-occurrence weighted mask;
determining, based on the joint probabilities, whether to reduce the number of candidate output tokens included in each combination of the one or more combinations;
identifying, based on the joint probabilities, a combination of the one or more combinations as the output token sequence; and
outputting, to the AI assistance device, a response to the request, the response comprising the output token sequence.

### Claim 2 (depends on 1)

The method of claim 1, wherein the predicting of the plurality of candidate output tokens comprises:
predicting each output token of the plurality of candidate output tokens by a corresponding token head of a plurality of token heads in a forward pass of the trained machine learning model, each corresponding token head being configured to provide a candidate output token in a corresponding position of the output token sequence.

### Claim 3 (depends on 2)

The method of claim 2, wherein the predicting of each output token of the plurality of candidate output tokens comprises:
predicting, by each corresponding token head of the plurality of token heads of the trained machine learning model, a plurality of logits corresponding to the plurality of candidate output tokens;
selecting, for each corresponding token head of the plurality of token heads, a number of candidate output tokens from the target tokens based on probabilities of the plurality of logits;
determining the one or more combinations by grouping a candidate output token from each corresponding token head,
wherein the estimating of the joint probability distributions comprises calculating, for each combination of the one or more combinations, a joint probability of candidate output tokens associated with the corresponding combination, and
wherein the determining whether to reduce the number of candidate output tokens comprises determining whether at least one combination of the one or more combinations has a joint probability exceeding a probability threshold.

### Claim 4 (depends on 3)

The method of claim 3, wherein the identifying of the combination of the one or more combinations comprises:
identifying, based on determining that the combination exceeds the probability threshold, the combination as the output token sequence.

### Claim 5 (depends on 3)

The method of claim 3, further comprising, based on determining that the one or more combinations do not exceed the probability threshold:
determining new candidate token combinations comprising candidate output tokens from a portion of the plurality of token heads, the portion of the plurality of token heads excluding at least one token head;
calculating joint probabilities for the new candidate token combinations; and
determining whether the joint probabilities of the new candidate token combinations exceed the probability threshold.

### Claim 6 (depends on 1)

The method of claim 1, wherein the co-occurrence weighted mask is independent of the input token sequence.

### Claim 7 (depends on 3)

The method of claim 3, further comprising:
adapting the probability threshold based on the joint probability of the candidate output tokens using a binarization algorithm.

### Claim 8 (depends on 3)

The method of claim 3, wherein the determining of the one or more combinations comprises:
estimating a posterior probability of each combination of the one or more combinations using a bigram correlation function; and
eliminating one or more combinations from the one or more combinations based on the posterior probabilities of the one or more combinations.

### Claim 9 (depends on 2)

The method of claim 2, further comprising:
transferring, from a trained base model, weight information to a model stem and a first token head of a plurality of token heads of an untrained machine learning model;
initializing weight information of remaining token heads of the plurality of token heads with randomized information; and
training the model stem and the first token head at a first learning rate and training the remaining token heads at second learning rate to obtain the trained machine learning model,
wherein the second learning rate is greater than the first learning rate.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
receiving, from the AI assistance device, the target language generation speed; and
determining, based on the target language generation speed, a target number of output tokens to be included in the output token sequence,
wherein the predicting of the plurality of candidate output tokens comprises predicting the target number of output tokens by a corresponding number of token heads of a plurality of token heads of the trained machine learning model.

### Claim 11

An apparatus for performing multi-token prediction, comprising:
one or more processors comprising processing circuitry; and
memory storing instructions,
wherein the instructions, when executed by the one or more processors individually or collectively, cause the apparatus to:
receive, from an artificial intelligence (AI) assistance device, a request for an output token sequence that is subsequent to an input token sequence indicated by the request;
predict, by a trained machine learning model, a plurality of candidate output tokens, the trained machine learning model having been trained to simultaneously predict the plurality of candidate output tokens based on the input token sequence;
estimate joint probability distributions of one or more combinations of the plurality of candidate output tokens, each combination of the one or more combinations having a number of candidate output tokens that corresponds to a target language generation speed;
calculate joint probabilities of the one or more combinations by masking the joint probability distributions with a co-occurrence weighted mask;
determine, based on the joint probabilities, whether to reduce the number of candidate output tokens included in each combination of the one or more combinations;
identify, based on the joint probabilities, a combination of the one or more combinations as the output token sequence; and
output, to the AI assistance device, a response to the request, the response comprising the output token sequence.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
predict each output token of the plurality of candidate output tokens by a corresponding token head of a plurality of token heads in a forward pass of the trained machine learning model, each corresponding token head being configured to provide a candidate output token in a corresponding position of the output token sequence.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
predict, by each corresponding token head of the plurality of token heads of the trained machine learning model, a plurality of logits corresponding to the plurality of candidate output tokens;
select, for each corresponding token head of the plurality of token heads, a number of candidate output tokens from the target tokens based on probabilities of the plurality of logits;
determine the one or more combinations by grouping a candidate output token from each corresponding token head;
calculate, for each combination of the one or more combinations, a joint probability of candidate output tokens associated with the corresponding combination, and
determine whether at least one combination of the one or more combinations has a joint probability exceeding a probability threshold.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
identify, based on determining that the combination exceeds the probability threshold, the combination as the output token sequence.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein the instructions, when executed by the one or more processors, further cause the apparatus, based on a determination that the one or more combinations do not exceed the probability threshold, to:
determine new candidate token combinations comprising candidate output tokens from a portion of the plurality of token heads, the portion of the plurality of token heads excluding at least one token head;
calculate joint probabilities for the new candidate token combinations; and
determine whether the joint probabilities of the new candidate token combinations exceed the probability threshold.

### Claim 16 (depends on 13)

The apparatus of claim 13, wherein the co-occurrence weighted mask is independent of the input token sequence.

### Claim 17 (depends on 13)

The apparatus of claim 13, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
adapt the probability threshold based on the joint probability of the candidate output tokens using a binarization algorithm.

### Claim 18 (depends on 13)

The apparatus of claim 13, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
estimate a posterior probability of each combination of the one or more combinations using a bigram correlation function; and
eliminate one or more combinations from the one or more combinations based on the posterior probabilities of the one or more combinations.

### Claim 19

A system for performing multi-token prediction, comprising:
an artificial intelligence (AI) assistance device; and
an apparatus, comprising:
one or more processors comprising processing circuitry; and
memory storing instructions,
wherein the instructions, when executed by the one or more processors individually or collectively, cause the apparatus to:
receive, from the AI assistance device, a request for an output token sequence that is subsequent to an input token sequence indicated by the request;
predict, by a trained machine learning model, a plurality of candidate output tokens, the trained machine learning model having been trained to simultaneously predict the plurality of candidate output tokens based on the input token sequence;
estimate joint probability distributions of one or more combinations of the plurality of candidate output tokens, each combination of the one or more combinations having a number of candidate output tokens that corresponds to a target language generation speed;
calculate joint probabilities of the one or more combinations by masking the joint probability distributions with a co-occurrence weighted mask;
determine, based on the joint probabilities, whether to reduce the number of candidate output tokens included in each combination of the one or more combinations;
identify, based on the joint probabilities, a combination of the one or more combinations as the output token sequence; and
output, to the AI assistance device, a response to the request, the response comprising the output token sequence.

### Claim 20 (depends on 19)

The system of claim 19, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
predict each output token of the plurality of candidate output tokens by a corresponding token head of a plurality of token heads in a forward pass of the trained machine learning model, each corresponding token head being configured to provide a candidate output token in a corresponding position of the output token sequence.
