# US 2025/0384243 A1 · Token pruning for language generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0384243 A1
- Document ID: US20250384243A1
- Status: Published application
- Kind: A1
- Application number: 18/742,242
- Filed: 2024-06-13
- Published: 2025-12-18
- Earliest claimed: 2024-06-13
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Haode Qi; Cheng Qian; Gengyu Wang; Sneha Srinivasan; Pratyush Singh; Ladislav Kunc; Juergen Bross; Jian Ni; Radu Florian
- Web page: https://patentgrep.com/patent/US20250384243A1

## CPC

- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Embodiments of the invention provide a computer-implemented method that includes executing, using a generative language model, generative language model operations operable to generate an output sequence responsive to an original input sequence. The generative language model operations include token pruning operations that include performing a base set of token pruning operations on intermediate versions of the original input sequence; and performing token pruning (TP) constraint evaluations. The base set of token pruning operations identify pruning candidate tokens in the intermediate versions of the original input sequence. The TP constraint evaluations determine that at least one of the pruning candidate tokens will be pruned from an associated intermediate version of the original input sequence.

## Claims

### Claim 1

A computer-implemented method comprising:
executing, using a generative language model, generative language model operations operable to generate an output sequence responsive to an original input sequence;
wherein the generative language model operations comprise token pruning operations comprising:
performing a base set of token pruning operations on intermediate versions of the original input sequence; and
performing token pruning (TP) constraint evaluations;
wherein the base set of token pruning operations identify pruning candidate tokens in the intermediate versions of the original input sequence; and
wherein the TP constraint evaluations determine that at least one of the pruning candidate tokens will be pruned from an associated intermediate version of the original input sequence.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the TP constraint evaluations determine that at least one of the pruning candidate tokens will not be pruned from the associated intermediate version of the original input sequence.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein:
the generative language model comprises transformer layers;
the transformer layers comprise a first transformer layer and a second transformer layer; and
the intermediate versions of the original input sequence comprises a first intermediate version of the original input sequence.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the token pruning operations are applied to the first intermediate version of the original input sequence before the first intermediate version of the original input sequence is passed from the first transformer layer to the second transformer layer.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the base set of token pruning operations identifying pruning candidate tokens in the intermediate versions of the original input sequence comprises determining importance level values for each token in the first intermediate version of the original input sequence.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the base set of token pruning operations identifying pruning candidate tokens in the intermediate versions of the original input sequence comprises comparing the importance level values of each token in the first intermediate version of the original input sequence to an importance level threshold.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the importance level value is based at least in part on an attention score.

### Claim 8

A computer system comprising:
a processor system and a memory electronically coupled to the processor system, wherein the memory stores a generative language model operable to perform generative language model operations that generate an output sequence responsive to an original input sequence;
wherein the generative language model operations comprise token pruning operations comprising:
performing a base set of token pruning operations on intermediate versions of the original input sequence; and
performing token pruning (TP) constraint evaluations;
wherein the base set of token pruning operations identify pruning candidate tokens in the intermediate versions of the original input sequence; and
wherein the TP constraint evaluations determine that at least one of the pruning candidate tokens will be pruned from an associated intermediate version of the original input sequence.

### Claim 9 (depends on 8)

The computer system of claim 8, wherein the TP constraint evaluations determine that at least one of the pruning candidate tokens will not be pruned from the associated intermediate version of the original input sequence.

### Claim 10 (depends on 8)

The computer system of claim 8, wherein:
the generative language model comprises transformer layers;
the transformer layers comprise a first transformer layer and a second transformer layer; and
the intermediate versions of the original input sequence comprises a first intermediate version of the original input sequence.

### Claim 11 (depends on 10)

The computer system of claim 10, wherein the token pruning operations are applied to the first intermediate version of the original input sequence before the first intermediate version of the original input sequence is passed from the first transformer layer to the second transformer layer.

### Claim 12 (depends on 11)

The computer system of claim 11, wherein the base set of token pruning operations identifying pruning candidate tokens in the intermediate versions of the original input sequence comprises determining importance level values for each token in the first intermediate version of the original input sequence.

### Claim 13 (depends on 12)

The computer system of claim 12, wherein the base set of token pruning operations identifying pruning candidate tokens in the intermediate versions of the original input sequence comprises comparing the importance level values of each token in the first intermediate version of the original input sequence to an importance level threshold.

### Claim 14 (depends on 13)

The computer system of claim 13, wherein the importance level value is based at least in part on an attention score.

### Claim 15

A computer program product comprising a computer readable storage medium storing a generative language model operable to perform generative language model operations that generate an output sequence responsive to an original input sequence;
wherein the generative language model operations comprise token pruning operations comprising:
performing a base set of token pruning operations on intermediate versions of the original input sequence; and
performing token pruning (TP) constraint evaluations;
wherein the base set of token pruning operations identify pruning candidate tokens in the intermediate versions of the original input sequence; and
wherein the TP constraint evaluations determine that at least one of the pruning candidate tokens will be pruned from an associated intermediate version of the original input sequence.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the TP constraint evaluations determine that at least one of the pruning candidate tokens will not be pruned from the associated intermediate version of the original input sequence.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein:
the generative language model comprises transformer layers;
the transformer layers comprise a first transformer layer and a second transformer layer;
the intermediate versions of the original input sequence comprises a first intermediate version of the original input sequence; and
the token pruning operations are applied to the first intermediate version of the original input sequence before the first intermediate version of the original input sequence is passed from the first transformer layer to the second transformer layer.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein the base set of token pruning operations identifying pruning candidate tokens in the intermediate versions of the original input sequence comprises determining importance level values for each token in the first intermediate version of the original input sequence.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein the base set of token pruning operations identifying pruning candidate tokens in the intermediate versions of the original input sequence comprises comparing the importance level values of each token in the first intermediate version of the original input sequence to an importance level threshold.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein the importance level value is based at least in part on an attention score.
