Differential quantization of a neural network based on svd information ranking

A method of operating a differentially quantized neural network comprising:

• receiving, at a first layer of the differentially quantized neural network, an input;
• performing, at the first layer, a first plurality of matrix operations using a significant quantized matrix and the input at a first precision level to produce a first result;
• performing, at the first layer, a second plurality of matrix operations using a less significant quantized matrix and the input at a second precision level to produce a second result, wherein the second precision level is lower than the first precision level;
• and
• combining the first result and the second result to produce an output for the first layer.

A method training a differentially quantized neural network, comprising:

• training a neural network at a full precision quantization to form a partially trained neural network;
• decomposing a weight matrix for a first layer of the partially trained neural network into an orthogonal matrix of singular values;
• using the orthogonal matrix to identify a first row or column of matrix values that makes a largest contribution to performance of the partially trained neural network;
• forming a significant matrix using the first row or column of matrix values;
• forming a less significant matrix using a second plurality of columns and/or rows from the orthogonal matrix that does not include first row or column of matrix values;
• quantizing the significant matrix at a first precision level to form a significant quantized matrix;
• quantizing the less significant matrix at a second precision level to form a less significant quantized matrix;
• including the significant matrix and the less significant matrix to form a partially trained differentially quantized neural network;
• training the differentially quantized neural network to form a quantized neural network;
• and
• deploying the quantized neural network to a client device.