Accelerated model inference using compressed model weights
AssigneeMicrosoft Technology Licensing, LLC
InventorsEric Chris Wolfgang SOMMERLADE, Karthik VIJAYAN
Life of this patent
Estimating the term…
Abstract
Systems and methods for accelerated model inference are provided. In particular, a computing device may receive a prediction request from an application in a production environment, decompress a first set of compressed weights of a compressed model based on the prediction request, perform evaluation of the compressed model using the first set of decompressed weights while decompressing a next set of compressed weights of the compressed model, generate a prediction using the decompressed weights, and return the prediction to the application.
Claims20
A method for accelerated model inference, the method comprising:
- receiving, by a computing device, a prediction request from an application in a production environment;
- decompressing, by the computing device, a first set of compressed weights of a compressed model based on the prediction request;
- performing, by the computing device, evaluation of the compressed model using the first set of decompressed weights while decompressing a next set of compressed weights of the compressed model;
- generating, by the computing device, a prediction using the decompressed weights;
- and
- returning, by the computing device, the prediction to the application.
A computing device for accelerated model inference, the computing device comprising:
- a processor;
- and
- a memory having a plurality of instructions stored thereon that, when executed by the processor, causes the computing device to:
- receive a prediction request from an application in a production environment;
- decompress a first set of compressed weights of a compressed model based on the prediction request;
- perform evaluation of the compressed model using the first set of decompressed weights while decompressing a next set of compressed weights of the compressed model;
- generate a prediction using the decompressed weights;
- and
- return the prediction to the application.
A method for model compression, the method comprising:
- generating a model;
- training the model to determine weights of the model for optimizing model outputs;
- performing quantization of the model to reduce a number of bits required to represent each weight of the model;
- and
- applying run-length encoding to the quantized weights to further compress the model.
Dependencies are inferred from the claim text.