US 2021/0232890 A1Published application

Cursor-based adaptive quantization for deep neural networks

AssigneesBaidu USA LLC, Baidu.Com Times Technology (Beijing) Co., Ltd.

InventorsBaopu LI, Yanwen FAN, Zhiyu CHENG, Yingze BAO

Life of this patent

Estimating the term…

Loading patent…

Abstract

Deep neural networks (DNN) model quantization may be used to reduce storage and computation burdens by decreasing the bit width. Presented herein are novel cursor-based adaptive quantization embodiments. In embodiments, a multiple bits quantization mechanism is formulated as a differentiable architecture search (DAS) process with a continuous cursor that represents a possible quantization bit. In embodiments, the cursor-based DAS adaptively searches for a quantization bit for each layer. The DAS process may be accelerated via an alternative approximate optimization process, which is designed for mixed quantization scheme of a DNN model. In embodiments, a new loss function is used in the search process to simultaneously optimize accuracy and parameter size of the model. In a quantization step, the closest two integers to the cursor may be adopted as the bits to quantize the DNN together to reduce the quantization noise and avoid the local convergence problem.

Claims20

Dependent claims

A computer-implement method for reducing memory size of a neural network, which comprises a plurality of layers, the method comprising:

responsive to a stop condition not being reached, performing steps comprising:
inputting a training data batch, which was obtained from a training data set comprising input data and corresponding ground truth data, into the neural network to obtain a training data output;
determining a training data loss for the neural network given the training data output from the neural network and a loss function that comprises a loss component related to prediction accuracy of the neural network and a compression component related to memory size of parameter values of the neural network after quantization;
updating at least some of the parameter values of the neural network using the training data loss;
inputting a validation data batch, which was obtained from a validation data set comprising input data and corresponding ground truth data, into the neural network to obtain a validation data output;
determining a validation data loss for the neural network given the validation data output from the neural network and the loss function;
updating cursor values for at least some of the layers of the neural network using the validation data loss, in which a cursor value for a layer in the neural network is related to memory size for representing parameter values for the layer;
and
for each layer of at least some of the layers, quantizing parameter values of the layer of the neural network using two integers that bound the cursor value for that layer, which quantized parameter values are used in determining the compression component of the loss function;
and
responsive to a stop condition being reached:
outputting final cursor values for the at least some of the layers of the neural network, in which the final cursor value for a layer of the neural network is the bit size for representing parameter values for that layer of the neural network;
and
outputting a set of parameter values for the neural network.

11 more elements

A system comprising:

one or more processors;
and
a non-transitory computer-readable medium or media storing one or more sequences of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
responsive to a stop condition not being reached, performing steps comprising:
inputting a training data batch, which was obtained from a training data set comprising input data and corresponding ground truth data, into the neural network to obtain a training data output;
determining a training data loss for the neural network given the training data output from the neural network and a loss function that comprises a loss component related to prediction accuracy of the neural network and a compression component related to memory size of parameter values of the neural network after quantization;
updating at least some of the parameter values of the neural network using the training data loss;
inputting a validation data batch, which was obtained from a validation data set comprising input data and corresponding ground truth data, into the neural network to obtain a validation data output;
determining a validation data loss for the neural network given the validation data output from the neural network and the loss function;
updating cursor values for at least some of the layers of the neural network using the validation data loss, in which a cursor value for a layer in the neural network is related to memory size for representing parameter values for the layer;
and
for each layer of at least some of the layers, quantizing parameter values of the layer of the neural network using two integers that bound the cursor value for that layer, which quantized parameter values are used in determining the compression component of the loss function;
and
responsive to a stop condition being reached:
outputting final cursor values for the at least some of the layers of the neural network, in which the final cursor value for a layer of the neural network is the bit size for representing parameter values for that layer of the neural network;
and
outputting a set of parameter values for the neural network.

14 more elements

A non-transitory computer-readable medium or media storing one or more sequences of instructions which, when executed by one or more processors, causes steps to be performed comprising:

responsive to a stop condition not being reached, performing steps comprising:
inputting a training data batch, which was obtained from a training data set comprising input data and corresponding ground truth data, into the neural network to obtain a training data output;
determining a training data loss for the neural network given the training data output from the neural network and a loss function that comprises a loss component related to prediction accuracy of the neural network and a compression component related to memory size of parameter values of the neural network after quantization;
updating at least some of the parameter values of the neural network using the training data loss;
inputting a validation data batch, which was obtained from a validation data set comprising input data and corresponding ground truth data, into the neural network to obtain a validation data output;
determining a validation data loss for the neural network given the validation data output from the neural network and the loss function;
updating cursor values for at least some of the layers of the neural network using the validation data loss, in which a cursor value for a layer in the neural network is related to memory size for representing parameter values for the layer;
and
for each layer of at least some of the layers, quantizing parameter values of the layer of the neural network using two integers that bound the cursor value for that layer, which quantized parameter values are used in determining the compression component of the loss function;
and
responsive to a stop condition being reached:
outputting final cursor values for the at least some of the layers of the neural network, in which the final cursor value for a layer of the neural network is the bit size for representing parameter values for that layer of the neural network;
and
outputting a set of parameter values for the neural network.

11 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/966,834Filed 24 Sep 2019
Published: 29 Jul 2021
Term ends: Not estimatedThe source records don’t support an estimate.

Classification9 codes

G06N 3/04
Neural networks: architecture, e.g. interconnection topology
G06N 3/063
Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons: using electronic means
G06N 3/0464
Architecture, e.g. interconnection topology: convolutional networks [CNN, ConvNet]
G06N 3/0495
Architecture, e.g. interconnection topology: quantised networks; Sparse networks; Compressed networks

Family

US 12,039,427 B2Related patent
US 2021/0232890 A1This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key