# US 2025/0094864 A1 · Compression of machine learning models via sparsification and quantization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094864 A1
- Document ID: US20250094864A1
- Status: Published application
- Kind: A1
- Application number: 18/602,951
- Filed: 2024-03-12
- Published: 2025-03-20
- Earliest claimed: 2023-09-14
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Po-An Tsai; Geonhwa Jeong; Jeffrey Michael Pool
- Web page: https://patentgrep.com/patent/US20250094864A1

## CPC

- G06N 20/00: Machine learning
- G06N 3/045: Combinations of networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Machine learning is a process that learns a model from a given dataset, where the model can then be used to make a prediction about new data. In order to reduce the size, computation, and latency of a machine learning model, a compression technique can be employed which includes model sparsification and quantization. To limit the extent to which the quality of the model is impacted when uniformly applying sparsification and quantization to all values of the model, the present disclosure provides for a hybrid sparsification and quantization of the model.

## Claims

### Claim 1

A method, comprising:
at a device, compressing a machine learning model having a plurality of values to reduce at least one of a size of the machine learning model or computation requirements of the machine learning model, by:
processing the machine learning model to generate a plurality of sparse data structures including:
storing inlier values of the machine learning model in a first data structure, and
storing outlier values of the machine learning model in a second data structure,
wherein at least one of the first data structure or the second data structure has a structured sparse pattern; and
non-uniformly quantizing the machine learning model, including:
quantizing the first data structure storing the inlier values to a first bit width, and
quantizing the second data structure storing the outlier values to a second bit width that is different from the first bit width.

### Claim 2 (depends on 1)

The method of claim 1, wherein the inlier values and the outlier values are weights of the machine learning model.

### Claim 3 (depends on 1)

The method of claim 1, wherein the inlier values and the outlier values are determined according to a defined threshold metric.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first data structure has a first structured sparse pattern that has less sparsity than a second structured sparse pattern of the second data structure.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first data structure has a first structured sparse pattern that is the same as a second structured sparse pattern of the second data structure.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first bit width and the second bit width are supported by different hardware accelerators.

### Claim 7

A method, comprising:
at a device:
apportioning a plurality of different subsets of values of the machine learning model into a plurality of data structures at least one of which has a defined structured sparse pattern; and
changing a data representation of at least one data structure of the plurality of data structures, wherein at least two data structures of the plurality of data structures have different data representations.

### Claim 8 (depends on 7)

The method of claim 7, wherein the machine learning model is a deep neural network.

### Claim 9 (depends on 7)

The method of claim 7, wherein the machine learning model is a large language model (LLM).

### Claim 10 (depends on 7)

The method of claim 7, wherein the values of the machine learning model are weights of the machine learning model.

### Claim 11 (depends on 7)

The method of claim 7, wherein the plurality of data structures are tensors.

### Claim 12 (depends on 7)

The method of claim 7, wherein at least two of the plurality of data structures have different defined structured sparse patterns.

### Claim 13 (depends on 12)

The method of claim 12, wherein the different defined structured sparse patterns include at least:
a first defined structured sparse pattern having a first sparsity degree, and
a second defined structured sparse pattern having a second sparsity degree,
wherein the first sparsity degree is different from the second sparsity degree.

### Claim 14 (depends on 7)

The method of claim 7, wherein the plurality of different subsets of values of the machine learning model include:
at least one subset comprised of at least a portion of inlier values of the machine learning model, and
at least another subset comprised of at least a portion of outlier values of the machine learning model.

### Claim 15 (depends on 14)

The method of claim 14, wherein the inlier values and the outlier values are determined according to a defined threshold metric.

### Claim 16 (depends on 15)

The method of claim 15, wherein the defined threshold metric is a magnitude of weight.

### Claim 17 (depends on 15)

The method of claim 15, wherein the defined threshold metric is an error after quantization for inlier and outlier.

### Claim 18 (depends on 15)

The method of claim 15, wherein the defined threshold metric is a product of a corresponding weight and activation.

### Claim 19 (depends on 14)

The method of claim 14, wherein at least a portion of the inlier values are stored with a first structured sparse pattern that has less sparsity than a second structured sparse pattern used to store at least a portion of the outlier values.

### Claim 20 (depends on 7)

The method of claim 7, wherein the machine learning model is further compressed by:
sparsifying the machine learning model by pruning values from the machine learning model, to form a sparse machine learning model,
wherein the plurality of different subsets of values of the machine learning model are determined from the sparse machine learning model.

### Claim 21 (depends on 20)

The method of claim 20, wherein the values of the machine learning model are selected for pruning according to a defined threshold metric.

### Claim 22 (depends on 21)

The method of claim 21, wherein the defined threshold metric is a magnitude of weight.

### Claim 23 (depends on 21)

The method of claim 21, wherein the defined threshold metric is an error after pruning.

### Claim 24 (depends on 21)

The method of claim 21, wherein the defined threshold metric is a product of a corresponding weight and activation obtained with training or validation data.

### Claim 25 (depends on 20)

The method of claim 20, wherein the machine learning model is sparsified to a defined degree of sparsity.

### Claim 26 (depends on 20)

The method of claim 20, wherein the machine learning model is sparsified with a defined structured sparse pattern.

### Claim 27 (depends on 7)

The method of claim 7, wherein changing the data representation of the at least one data structure includes quantizing the at least one data structure.

### Claim 28 (depends on 7)

The method of claim 7, wherein the data representation of the plurality of data structures includes a bit width of the plurality of data structures.

### Claim 29 (depends on 7)

The method of claim 7, wherein the data representation of the plurality of data structures includes a data type of the plurality of data structures.

### Claim 30 (depends on 7)

The method of claim 7, wherein the different data representations are supported by a single hardware accelerator or multiple different hardware accelerators.

### Claim 31 (depends on 7)

The method of claim 7, wherein at least two of the plurality of data structures have different defined structured sparse patterns, and wherein the different defined structured sparse patterns are supported by a single hardware accelerator or multiple different hardware accelerators.

### Claim 32

A system, comprising:
a non-transitory memory storage comprising instructions; and
one or more processors in communication with the memory, wherein the one or more processors execute the instructions to at least one of compress a machine learning model or reduce a computation of the machine learning model by:
apportioning a plurality of different subsets of values of the machine learning model into a plurality of data structures at least one of which has a defined structured sparse pattern; and
changing a data representation of at least one data structure of the plurality of data structures, wherein at least two data structures of the plurality of data structures have different data representations.

### Claim 33 (depends on 32)

The system of claim 32, wherein the machine learning model is a deep neural network.

### Claim 34 (depends on 32)

The system of claim 32, wherein the machine learning model is a large language model (LLM).

### Claim 35 (depends on 32)

The system of claim 32, wherein the values of the machine learning model are weights of the machine learning model.

### Claim 36 (depends on 32)

The system of claim 32, wherein at least two of the plurality of data structures have different defined structured sparse patterns.

### Claim 37 (depends on 32)

The system of claim 32, wherein the plurality of different subsets of values of the machine learning model include:
at least one subset comprised of at least a portion of inlier values of the machine learning model, and
at least another subset comprised of at least a portion of outlier values of the machine learning model.

### Claim 38 (depends on 37)

The system of claim 37, wherein at least a portion of the inlier values are stored with a first structured sparse pattern that has less sparsity than a second structured sparse pattern used to store at least a portion of the outlier values.

### Claim 39 (depends on 32)

The system of claim 32, wherein changing the data representation of the at least one data structure includes quantizing the at least one data structure.

### Claim 40 (depends on 32)

The system of claim 32, wherein the data representation of the plurality of data structures includes a bit width of the plurality of data structures.

### Claim 41 (depends on 32)

The system of claim 32, wherein the data representation of the plurality of data structures includes a data type of the plurality of data structures.

### Claim 42

A non-transitory computer-readable media storing computer instructions which when executed by one or more processors of a device cause the device to at least one of compress a machine learning model or reduce a computation of the machine learning model by:
apportioning a plurality of different subsets of values of the machine learning model into a plurality of data structures at least one of which has a defined structured sparse pattern; and
changing a data representation of at least one data structure of the plurality of data structures, wherein at least two data structures of the plurality of data structures have different data representations.

### Claim 43 (depends on 42)

The non-transitory computer-readable media of claim 42, wherein the machine learning model is a deep neural network.

### Claim 44 (depends on 42)

The non-transitory computer-readable media of claim 42, wherein the machine learning model is a large language model (LLM).

### Claim 45 (depends on 42)

The non-transitory computer-readable media of claim 42, wherein the values of the machine learning model are weights of the machine learning model.

### Claim 46 (depends on 42)

The non-transitory computer-readable media of claim 42, wherein at least two of the plurality of data structures have different defined structured sparse patterns.

### Claim 47 (depends on 42)

The non-transitory computer-readable media of claim 42, wherein the plurality of different subsets of values of the machine learning model include:
at least one subset comprised of at least a portion of inlier values of the machine learning model, and
at least another subset comprised of at least a portion of outlier values of the machine learning model.

### Claim 48 (depends on 47)

The non-transitory computer-readable media of claim 47, wherein at least a portion of the inlier values are stored with a first structured sparse pattern that has less sparsity than a second structured sparse pattern used to store at least a portion of the outlier values.

### Claim 49 (depends on 42)

The non-transitory computer-readable media of claim 42, wherein changing the data representation of the at least one data structure includes quantizing the at least one data structure.

### Claim 50 (depends on 42)

The non-transitory computer-readable media of claim 42, wherein the data representation of the plurality of data structures includes a bit width of the plurality of data structures.

### Claim 51 (depends on 42)

The non-transitory computer-readable media of claim 42, wherein the data representation of the plurality of data structures includes a data type of the plurality of data structures.
