US 2026/0073237 A1Published application

Quantization-Based Method and System for Federated Fine-Tuning a Pre-Trained Large Language Model

AssigneeCity University of Hong Kong

InventorsDapeng WU, Hong HUANG

Life of this patent

Estimating the term…

Loading patent…

Abstract

The present invention provides a quantization-based method and system for federated fine-tuning a pre-trained large language model (LLM). The system comprises an accelerator configured to identify outlier channels of the LLM and inject a set of adapters into the LLM; and local devices configured to: construct channel-wise scaling vectors on basis of the indices of the identified outlier channels; apply the channel-wise scaling vectors on input matrices of the LLM to obtain scaled activation matrices; quantize the scaled activation matrices to obtain quantized activation matrices; apply the channel-wise scaling vectors on the outlier weight matrices of the LLM to obtain scaled outlier weight matrices; quantize the scaled outlier weight matrices to obtain quantized outlier weight matrices; fine-tune the adapters of the LLM with the quantized activation matrices and the quantized outlier weight matrices. The server is further configured to perform weight aggregation to produce new adapters for the LLM.

Claims20

Dependent claims

A quantization-based method for federated fine-tuning a pre-trained large language model with a server and one or more devices, comprising:

(a) freezing, by the server, all weight matrices of the pre-trained language model;
(b) identifying, by the server, one or more outlier channels from a plurality of channels of the large language model on the basis of public datasets;
(c) extracting, by the server, outlier weight matrices on the basis of indices of the identified outlier channels from the frozen weight matrices of the pre-trained language model;
(d) quantizing, by the server, the frozen weight matrices of the pre-trained large language model;
(e) injecting, by the server, a set of trainable parameters into the pre-trained language model;
(f) downloading, by each device, the quantized weight matrices of the pre-trained large language model, the outlier weight matrices of the pre-trained large language model and the indices of the identified outlier channels from the server;
(g) downloading, by each device, the adapters of the pre-trained large language model;
(h) constructing, by each device, channel-wise scaling vectors on basis of the indices of the identified outlier channels;
(i) applying, by each device, the channel-wise scaling vectors on input matrices of the pre-trained large language model to obtain scaled activation matrices;
(j) quantizing, by each device, the scaled activation matrices to obtain quantized activation matrices;
(k) applying, by each device, the channel-wise scaling vectors on the outlier weight matrices of the pre-trained large language model to obtain scaled outlier weight matrices;
(l) quantizing, by each device, the scaled outlier weight matrices to obtain quantized outlier weight matrices;
(m) fine-tuning, by each device, the adapters of the large language model with the quantized activation matrices and the quantized outlier weight matrices;
(n) uploading, by each device, the adapters of the large language model to the server;
(o) performing, by the server, weight aggregation to produce new adapters for the large language model;
(p) repeating steps (g) to (o) until a stop criterion is met.

13 more elements

A quantization-based system for federated fine-tuning a pre-trained large language model, comprises:

a memory configured to store the large language model and input matrices;
an accelerator configured to:
freeze all weight matrices of the pre-trained language model;
identify one or more outlier channels from a plurality of channels of the large language model on the basis of public datasets;
extract outlier weight matrices on the basis of indices of the identified outlier channels from the frozen weight matrices of the pre-trained language model;
quantize the frozen weight matrices of the pre-trained large language model;
and
inject a set of adapters into the pre-trained language model;
and
one or more local devices, each device is configured to:
download the quantized weight matrices of the pre-trained large language model, outlier weight matrices of the pre-trained large language model, the adapters of the pre-trained large language model and the indices of the identified outlier channels from the server;
construct channel-wise scaling vectors on basis of the indices of the identified outlier channels;
apply the channel-wise scaling vectors on input matrices of the pre-trained large language model to obtain scaled activation matrices;
quantize the scaled activation matrices to obtain quantized activation matrices;
apply the channel-wise scaling vectors on the outlier weight matrices of the pre-trained large language model to obtain scaled outlier weight matrices;
quantize the scaled outlier weight matrices to obtain quantized outlier weight matrices;
fine-tune the adapters of the large language model with the quantized activation matrices and the quantized outlier weight matrices;
and
upload the adapters of the large language model to the server;
and
wherein the server is further configured to perform weight aggregation to produce new adapters for the large language model.

18 more elements

Dependencies are inferred from the claim text.