# US 2025/0384221 A1 · Compression of models for natural language processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0384221 A1
- Document ID: US20250384221A1
- Status: Published application
- Kind: A1
- Application number: 19/307,850
- Filed: 2025-08-22
- Published: 2025-12-18
- Earliest claimed: 2021-08-12
- Assignees: Wells Fargo Bank, N.A.
- Applicants: Wells Fargo Bank, N.A.
- Inventors: Tarun Joshi; Archit Parnami; Rahul Singh
- Web page: https://patentgrep.com/patent/US20250384221A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An example electronic computing device can include: a processor; and a system memory, the system memory including instructions which, when executed by the processor, cause the electronic computing device to: receive a model for natural language processing of data, the model including a plurality of self-attention heads; prune the model by removing one or more of the plurality of self-attention heads of the model to create a pruned model; and evaluate a classification accuracy of the pruned model to maintain a performance level.

## Claims

### Claim 1

An electronic computing device, comprising:
a processor; and
a system memory, the system memory including instructions which, when executed by the processor, cause the electronic computing device to:
receive a model for natural language processing of data, the model including self-attention layers having multiple self-attention heads;
identify a performance baseline for the model based on an initial classification accuracy;
sequentially remove individual self-attention heads from different self-attention layers of the model to create a pruned model;
evaluate a classification accuracy of the pruned model after removal of the individual self-attention heads; and
determine a final set of self-attention heads to be pruned based on comparing the classification accuracy after removal of the individual self-attention heads to the performance baseline.

### Claim 2 (depends on 1)

The electronic computing device of claim 1, wherein the model is a Bidirectional Encoder Representations from Transformers model or a Generative Pre-trained Transformer model.

### Claim 3 (depends on 1)

The electronic computing device of claim 1, further comprising instructions which, when executed by the processor, cause the electronic computing device to train the model using a given training dataset.

### Claim 4 (depends on 1)

The electronic computing device of claim 1, further comprising instructions which, when executed by the processor, cause the electronic computing device to use an A* algorithm to prune the model.

### Claim 5 (depends on 4)

The electronic computing device of claim 4, wherein the A* algorithm is a search heuristic algorithm.

### Claim 6 (depends on 1)

The electronic computing device of claim 1, further comprising instructions which, when executed by the processor, cause the electronic computing device to calculate a performance cost associated with removing each individual self-attention head, wherein the performance cost represents a drop in the classification accuracy from the performance baseline.

### Claim 7 (depends on 1)

The electronic computing device of claim 1, further comprising instructions which, when executed by the processor, cause the electronic computing device to define a budget that quantifies a maximum amount of classification accuracy that can be sacrificed during pruning.

### Claim 8 (depends on 7)

The electronic computing device of claim 7, wherein the budget defines a boundary for the classification accuracy, and wherein sequential removal of the individual self-attention heads continues until the budget is exceeded.

### Claim 9 (depends on 1)

The electronic computing device of claim 1, further comprising instructions which, when executed by the processor, cause the electronic computing device to sort the individual self-attention heads in ascending order based on their respective performance costs before determining the final set of self-attention heads to be pruned.

### Claim 10 (depends on 1)

The electronic computing device of claim 1, further comprising instructions which, when executed by the processor, cause the electronic computing device to iteratively prune self-attention heads from multiple self-attention layers, wherein each iteration removes a single self-attention head with a lowest performance cost from among remaining unpruned self-attention heads.

### Claim 11

A method, comprising:
receiving a model for natural language processing of data, the model including self-attention layers having multiple self-attention heads;
identifying a performance baseline for the model based on an initial classification accuracy;
sequentially removing individual self-attention heads from different self-attention layers of the model to create a pruned model;
evaluating a classification accuracy of the pruned model after removal of the individual self-attention heads; and
determining a final set of self-attention heads to be pruned based on comparing the classification accuracy after removal of the individual self-attention heads to the performance baseline.

### Claim 12 (depends on 11)

The method of claim 11, wherein the model is a Bidirectional Encoder Representations from Transformers model or a Generative Pre-trained Transformer model.

### Claim 13 (depends on 11)

The method of claim 11, further comprising training the model using a given training dataset.

### Claim 14 (depends on 11)

The method of claim 11, further comprising using an A* algorithm to prune the model.

### Claim 15 (depends on 14)

The method of claim 14, wherein the A* algorithm is a search heuristic algorithm.

### Claim 16 (depends on 11)

The method of claim 11, further comprising calculating a performance cost associated with removing each individual self-attention head, wherein the performance cost represents a drop in the classification accuracy from the performance baseline.

### Claim 17 (depends on 11)

The method of claim 11, further comprising defining a budget that quantifies a maximum amount of classification accuracy that can be sacrificed during pruning.

### Claim 18 (depends on 17)

The method of claim 17, wherein the budget defines a boundary for the classification accuracy, and wherein sequential removal of the individual self-attention heads continues until the budget is exceeded.

### Claim 19 (depends on 11)

The method of claim 11, further comprising sorting the individual self-attention heads in ascending order based on their respective performance costs before determining the final set of self-attention heads to be pruned.

### Claim 20 (depends on 11)

The method of claim 11, further comprising iteratively pruning self-attention heads from multiple self-attention layers, wherein each iteration removes a single self-attention head with a lowest performance cost from among remaining unpruned self-attention heads.
