# US 2026/0073237 A1 · Quantization-Based Method and System for Federated Fine-Tuning a Pre-Trained Large Language Model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0073237 A1
- Document ID: US20260073237A1
- Status: Published application
- Kind: A1
- Application number: 18/826,231
- Filed: 2024-09-06
- Published: 2026-03-12
- Earliest claimed: 2024-09-06
- Assignees: City University of Hong Kong
- Applicants: City University of Hong Kong
- Inventors: Dapeng WU; Hong HUANG
- Web page: https://patentgrep.com/patent/US20260073237A1

## CPC

- G06N 3/098: Distributed learning, e.g. federated learning
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present invention provides a quantization-based method and system for federated fine-tuning a pre-trained large language model (LLM). The system comprises an accelerator configured to identify outlier channels of the LLM and inject a set of adapters into the LLM; and local devices configured to: construct channel-wise scaling vectors on basis of the indices of the identified outlier channels; apply the channel-wise scaling vectors on input matrices of the LLM to obtain scaled activation matrices; quantize the scaled activation matrices to obtain quantized activation matrices; apply the channel-wise scaling vectors on the outlier weight matrices of the LLM to obtain scaled outlier weight matrices; quantize the scaled outlier weight matrices to obtain quantized outlier weight matrices; fine-tune the adapters of the LLM with the quantized activation matrices and the quantized outlier weight matrices. The server is further configured to perform weight aggregation to produce new adapters for the LLM.

## Claims

### Claim 1

A quantization-based method for federated fine-tuning a pre-trained large language model with a server and one or more devices, comprising:
(a) freezing, by the server, all weight matrices of the pre-trained language model;
(b) identifying, by the server, one or more outlier channels from a plurality of channels of the large language model on the basis of public datasets;
(c) extracting, by the server, outlier weight matrices on the basis of indices of the identified outlier channels from the frozen weight matrices of the pre-trained language model;
(d) quantizing, by the server, the frozen weight matrices of the pre-trained large language model;
(e) injecting, by the server, a set of trainable parameters into the pre-trained language model;
(f) downloading, by each device, the quantized weight matrices of the pre-trained large language model, the outlier weight matrices of the pre-trained large language model and the indices of the identified outlier channels from the server;
(g) downloading, by each device, the adapters of the pre-trained large language model;
(h) constructing, by each device, channel-wise scaling vectors on basis of the indices of the identified outlier channels;
(i) applying, by each device, the channel-wise scaling vectors on input matrices of the pre-trained large language model to obtain scaled activation matrices;
(j) quantizing, by each device, the scaled activation matrices to obtain quantized activation matrices;
(k) applying, by each device, the channel-wise scaling vectors on the outlier weight matrices of the pre-trained large language model to obtain scaled outlier weight matrices;
(l) quantizing, by each device, the scaled outlier weight matrices to obtain quantized outlier weight matrices;
(m) fine-tuning, by each device, the adapters of the large language model with the quantized activation matrices and the quantized outlier weight matrices;
(n) uploading, by each device, the adapters of the large language model to the server;
(o) performing, by the server, weight aggregation to produce new adapters for the large language model;
(p) repeating steps (g) to (o) until a stop criterion is met.

### Claim 2 (depends on 1)

The quantization-based method according to claim 1, wherein one or more outlier channels are identified by:
determining, for each channel of the large language model, whether data points in the channel include one or more outlier data points; and
identifying the channel as an outlier channel if the data points in the corresponding channel include outlier data points.

### Claim 3 (depends on 2)

The quantization-based method according to claim 2, wherein one or more outlier data points are data points having absolute values greater than an outlier threshold corresponding to the corresponding channel.

### Claim 4 (depends on 3)

The quantization-based method according to claim 3, wherein the outlier threshold is dynamically determined on the basis of the data points in the corresponding channel.

### Claim 5 (depends on 4)

The quantization-based method according to claim 4, wherein the outlier threshold is determined on the basis of an interquartile range of the data points in the corresponding channel.

### Claim 6 (depends on 4)

The quantization-based method according to claim 4, wherein the outlier threshold is determined on the basis of an average of absolute values of the data points in the corresponding channel.

### Claim 7 (depends on 1)

The quantization-based method according to claim 1, wherein the channel-wise scaling vector is constructed on basis of the indices of the identified outlier channels by:
setting a value of an entry of the channel-wise scaling vector equal to one if an index of the entry does not correspond to a position number of an outlier channel; and
setting a value of an entry of the channel-wise scaling vector equal to a scaling factor if an index of the entry corresponds to a position number of an outlier channel.

### Claim 8 (depends on 1)

The quantization-based method according to claim 1, wherein the scaling factor is computed on the basis of a maximum element of the input matrix and a maximum element of the weight matrix.

### Claim 9 (depends on 1)

The quantization-based method according to claim 1, wherein the plurality of channels respectively correspond to plurality formats for expressing semantic information defined under the large language model.

### Claim 10 (depends on 9)

The quantization-based method according to claim 9, wherein the large language model is fine-tuned for question answering, text summation and/or sentiment analysis.

### Claim 11

A quantization-based system for federated fine-tuning a pre-trained large language model, comprises:
a memory configured to store the large language model and input matrices;
an accelerator configured to:
freeze all weight matrices of the pre-trained language model;
identify one or more outlier channels from a plurality of channels of the large language model on the basis of public datasets;
extract outlier weight matrices on the basis of indices of the identified outlier channels from the frozen weight matrices of the pre-trained language model;
quantize the frozen weight matrices of the pre-trained large language model; and
inject a set of adapters into the pre-trained language model; and
one or more local devices, each device is configured to:
download the quantized weight matrices of the pre-trained large language model, outlier weight matrices of the pre-trained large language model, the adapters of the pre-trained large language model and the indices of the identified outlier channels from the server;
construct channel-wise scaling vectors on basis of the indices of the identified outlier channels;
apply the channel-wise scaling vectors on input matrices of the pre-trained large language model to obtain scaled activation matrices;
quantize the scaled activation matrices to obtain quantized activation matrices;
apply the channel-wise scaling vectors on the outlier weight matrices of the pre-trained large language model to obtain scaled outlier weight matrices;
quantize the scaled outlier weight matrices to obtain quantized outlier weight matrices;
fine-tune the adapters of the large language model with the quantized activation matrices and the quantized outlier weight matrices; and
upload the adapters of the large language model to the server; and
wherein the server is further configured to perform weight aggregation to produce new adapters for the large language model.

### Claim 12 (depends on 11)

The quantization-based system according to claim 11, wherein one or more outlier channels are identified by:
determining, for each channel of the large language model, whether data points in the channel include one or more outlier data points; and
identifying the channel as an outlier channel if the data points in the corresponding channel include outlier data points.

### Claim 13 (depends on 12)

The quantization-based system according to claim 12, wherein one or more outlier data points are data points having absolute values greater than an outlier threshold corresponding to the corresponding channel.

### Claim 14 (depends on 13)

The quantization-based system according to claim 13, wherein the outlier threshold is dynamically determined on the basis of the data points in the corresponding channel.

### Claim 15 (depends on 14)

The quantization-based system according to claim 14, wherein the outlier threshold is determined on the basis of an interquartile range of the data points in the corresponding channel.

### Claim 16 (depends on 14)

The quantization-based system according to claim 14, wherein the outlier threshold is determined on the basis of an average of absolute values of the data points in the corresponding channel.

### Claim 17 (depends on 11)

The quantization-based system according to claim 11, wherein the channel-wise scaling vector is constructed on basis of the indices of the identified outlier channels by:
setting a value of an entry of the channel-wise scaling vector equal to one if an index of the entry corresponds to a position number of an outlier channel; and
setting a value of an entry of the channel-wise scaling vector equal to a scaling factor if an index of the entry does not correspond to a position number of an outlier channel.

### Claim 18 (depends on 11)

The quantization-based system according to claim 11, wherein the scaling factor is computed on the basis of a maximum element of the input matrix and a maximum element of the weight matrix.

### Claim 19 (depends on 11)

The quantization-based system according to claim 11, wherein the plurality of channels respectively correspond to a plurality of formats for expressing semantic information defined under the large language model.

### Claim 20 (depends on 19)

The quantization-based system according to claim 19, wherein the large language model is fine-tuned for question answering, text summation and/or sentiment analysis.
