Dynamic transformer layers
AssigneeIntuit Inc.
InventorsMatan VETZLER, Shai ARDAZI, Kfir AHARON, Linoy COHEN
Life of this patent
Estimating the term…
Abstract
A system and method for dynamic layer quantization in large language models (LLMs) is described. A classifier predicts quantization levels for LLM layers based on input data and current layer output. The system applies these predicted levels to subsequent layers, allowing efficient LLM operation across various devices. This approach may help balance performance and resource use. The system can use federated learning to improve both the LLM and the classifier, potentially adapting better to different devices and uses.
Claims20
A method for dynamic layer quantization in a language model, comprising:
- receiving, at a classifier comprising at least one processor, query data from a user device and output data from a current layer of a large language model (LLM);
- predicting, by the classifier, a quantization level for a subsequent layer of the LLM corresponding to the query data from the user device and output data from the current layer of the LLM;
- selecting, by the classifier, the predicted quantization level for the subsequent layer of the LLM;
- processing, by the LLM, the output data of from the current layer of the LLM through the quantized subsequent layer of the LLM;
- and
- outputting, by the LLM, a response to the user query.
A system for dynamic layer quantization in a language model, comprising:
- a classifier comprising at least one processor, the classifier configured to:
- receive query data from a user device and output data from a current layer of a large language model (LLM);
- predict a quantization level for a subsequent layer of the LLM corresponding to the query data from the user device and output data from the current layer of the LLM;
- select the predicted quantization level for the subsequent layer of the LLM;
- the LLM configured to:
- process the output data from the current layer of the LLM through the quantized subsequent layer of the LLM;
- and
- output a response to the user query.
Dependencies are inferred from the claim text.