# US 2026/0087333 A1 · Accelerated model inference using compressed model weights

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087333 A1
- Document ID: US20260087333A1
- Status: Published application
- Kind: A1
- Application number: 18/937,802
- Filed: 2024-11-05
- Published: 2026-03-26
- Earliest claimed: 2024-09-26
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Eric Chris Wolfgang SOMMERLADE; Karthik VIJAYAN
- Web page: https://patentgrep.com/patent/US20260087333A1

## CPC

- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for accelerated model inference are provided. In particular, a computing device may receive a prediction request from an application in a production environment, decompress a first set of compressed weights of a compressed model based on the prediction request, perform evaluation of the compressed model using the first set of decompressed weights while decompressing a next set of compressed weights of the compressed model, generate a prediction using the decompressed weights, and return the prediction to the application.

## Claims

### Claim 1

A method for accelerated model inference, the method comprising:
receiving, by a computing device, a prediction request from an application in a production environment;
decompressing, by the computing device, a first set of compressed weights of a compressed model based on the prediction request;
performing, by the computing device, evaluation of the compressed model using the first set of decompressed weights while decompressing a next set of compressed weights of the compressed model;
generating, by the computing device, a prediction using the decompressed weights; and
returning, by the computing device, the prediction to the application.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving, by the computing device, a compressed model from a server, wherein the compressed model includes a model description including one or more parameters and attributes associated with the compressed model.

### Claim 3 (depends on 2)

The method of claim 2, wherein the one or more parameters and attributes indicate how nodes or layers of the model are connected and a sequential order of which a set of compressed weights of the compressed model to be decompressed for execution.

### Claim 4 (depends on 3)

The method of claim 3, wherein the decompressing the first set of compressed weights of the compressed based on the prediction request comprises:
extracting and decompressing, by the computing device, a portion of the compressed model based on the prediction request and the model description, wherein the portion of the compressed model defined by the first set of decompressed weights.

### Claim 5 (depends on 3)

The method of claim 3, wherein the next set of compressed weights of the compressed model is determined based on the model description associated with the compressed model.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
storing, by the computing device, the first set and/or the next set of decompressed weights on a cache memory.

### Claim 7 (depends on 1)

The method of claim 1, wherein the compressed model is stored in a compressed form in a main memory of the computing device during the accelerated model inference.

### Claim 8 (depends on 1)

The method of claim 1, wherein performing the evaluation of the compressed model comprises:
performing summation and scaling of the first set of decompressed weights that is mathematically equivalent to an innermost calculation of a dot product during a convolution operation.

### Claim 9

A computing device for accelerated model inference, the computing device comprising:
a processor; and
a memory having a plurality of instructions stored thereon that, when executed by the processor, causes the computing device to:
receive a prediction request from an application in a production environment;
decompress a first set of compressed weights of a compressed model based on the prediction request;
perform evaluation of the compressed model using the first set of decompressed weights while decompressing a next set of compressed weights of the compressed model;
generate a prediction using the decompressed weights; and
return the prediction to the application.

### Claim 10 (depends on 9)

The computing device of claim 9, wherein the plurality of instructions, when executed, further cause the computing device to:
receive a compressed model from a server, wherein the compressed model includes a model description including one or more parameters and attributes associated with the compressed model.

### Claim 11 (depends on 10)

The computing device of claim 10, wherein the one or more parameters and attributes indicate how nodes or layers of the model are connected and a sequential order of which a set of compressed weights of the compressed model to be decompressed for execution.

### Claim 12 (depends on 11)

The computing device of claim 11, wherein the decompressing the first set of compressed weights of the compressed based on the prediction request comprises:
extracting and decompressing, by the computing device, a portion of the compressed model based on the prediction request and the model description, wherein the portion of the compressed model defined by the first set of decompressed weights.

### Claim 13 (depends on 11)

The computing device of claim 11, wherein the next set of compressed weights of the compressed model is determined based on the model description associated with the compressed model.

### Claim 14 (depends on 9)

The computing device of claim 9, wherein the plurality of instructions, when executed, further cause the computing device to store the first set and/or the next set of decompressed weights on a cache memory.

### Claim 15 (depends on 9)

The computing device of claim 9, wherein the compressed model is stored in a compressed form in a main memory of the computing device during the accelerated model inference.

### Claim 16 (depends on 9)

The computing device of claim 9, wherein to perform the evaluation of the compressed model comprises to:
perform summation and scaling of the first set of decompressed weights that is mathematically equivalent to an innermost calculation of a dot product during a convolution operation.

### Claim 17

A method for model compression, the method comprising:
generating a model;
training the model to determine weights of the model for optimizing model outputs;
performing quantization of the model to reduce a number of bits required to represent each weight of the model; and
applying run-length encoding to the quantized weights to further compress the model.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
defining a model description associated with the model, the model description including one or more parameters and attributes associated with the compressed model.

### Claim 19 (depends on 18)

The method of claim 18, wherein the one or more parameters and attributes indicate how nodes or layers of the model are connected.

### Claim 20 (depends on 17)

The method of claim 17, further comprising shuffling the quantized weights to minimize an encoding dictionary and increase run length of indices.
