Back to results
US 2025/0384221 A1Published application
Compression of models for natural language processing
AssigneeWells Fargo Bank, N.A.
InventorsTarun Joshi, Archit Parnami, Rahul Singh
Life of this patent
Estimating the term…
Abstract
An example electronic computing device can include: a processor; and a system memory, the system memory including instructions which, when executed by the processor, cause the electronic computing device to: receive a model for natural language processing of data, the model including a plurality of self-attention heads; prune the model by removing one or more of the plurality of self-attention heads of the model to create a pruned model; and evaluate a classification accuracy of the pruned model to maintain a performance level.
Claims20
An electronic computing device, comprising:
- a processor;
- and
- a system memory, the system memory including instructions which, when executed by the processor, cause the electronic computing device to:
- receive a model for natural language processing of data, the model including self-attention layers having multiple self-attention heads;
- identify a performance baseline for the model based on an initial classification accuracy;
- sequentially remove individual self-attention heads from different self-attention layers of the model to create a pruned model;
- evaluate a classification accuracy of the pruned model after removal of the individual self-attention heads;
- and
- determine a final set of self-attention heads to be pruned based on comparing the classification accuracy after removal of the individual self-attention heads to the performance baseline.
6 more elements
A method, comprising:
- receiving a model for natural language processing of data, the model including self-attention layers having multiple self-attention heads;
- identifying a performance baseline for the model based on an initial classification accuracy;
- sequentially removing individual self-attention heads from different self-attention layers of the model to create a pruned model;
- evaluating a classification accuracy of the pruned model after removal of the individual self-attention heads;
- and
- determining a final set of self-attention heads to be pruned based on comparing the classification accuracy after removal of the individual self-attention heads to the performance baseline.
3 more elements
Dependencies are inferred from the claim text.