# US 2017/0286830 A1 · Quantized neural network training and inference

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286830 A1
- Document ID: US20170286830A1
- Status: Published application
- Kind: A1
- Application number: 15/478,531
- Filed: 2017-04-04
- Published: 2017-10-05
- Earliest claimed: 2016-04-04
- Assignees: Technion Research & Development Foundation Limited
- Applicants: Technion Research & Development Foundation Limited
- Inventors: Ran EL-YANIV; Itay HUBARA; Daniel SOUDRY
- Web page: https://patentgrep.com/patent/US20170286830A1

## CPC

- G06F 7/523: Multiplying only
- G06F 7/48: using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06F 2207/4824: Neural networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Training neural networks by constructing a neural network model having neurons each associated with a quantized activation function adapted to output a quantized activation value. The neurons are arranged in layers and connected by connections associated quantized connection weight functions adapted to output quantized connection weight values. During a training process a plurality of weight gradients are calculated during backpropagation sub-processes by computing neuron gradients, each of an output of a respective the quantized activation function in one layer with respect to an input of the respective quantized activation function. Each neuron gradient is calculated such that when an absolute value of the input is smaller than a positive constant threshold value, the respective neuron gradient is set as a positive constant output value and when the absolute value of the input is smaller than the positive constant threshold value the neuron gradient is set to zero.

## Claims

### Claim 1

A method for training neural networks, comprising:
constructing a neural network model having a plurality of neurons each associated with a quantized activation function adapted to output a quantized activation value selected from a first finite set, said plurality of neurons are arranged in a plurality of layers and being connected by a plurality of connections each associated with a quantized connection weight function adapted to output a quantized connection weight value selected from a second finite set;
receiving a training set dataset;
using said training set dataset to train said neural network model according to respective said quantized connection weight values, said training includes computing a plurality of weight gradients for backpropagation sub-processes by:
computing a plurality of neuron gradients, each said neuron gradient is of an output of a respective said quantized activation function in one layer of said plurality of layers with respect to an input of said respective quantized activation function and is calculated such that when an absolute value of said input is smaller than a positive constant threshold value, said respective neuron gradient is set as a positive constant value and when the absolute value of said input is larger than said positive constant threshold value said neuron gradient is set to zero, and
updating a plurality of floating point connection weight values according to said plurality of weight gradients;
outputting a trained quantized neural network formed as an outcome of said training process.

### Claim 2 (depends on 1)

The method of claim 1, wherein each of said plurality of connections is associated with one of said plurality of floating point connection weight values.

### Claim 3 (depends on 1)

The method of claim 1, wherein said neural network model is a convolutional neural network (CNN) model and said trained quantized neural network is a trained quantized CNN.

### Claim 4 (depends on 1)

The method of claim 1, wherein said quantized activation function is a binary function and said quantized activation value is a binary value.

### Claim 5 (depends on 1)

The method of claim 1, wherein said quantized connection weight function is a binary function and said quantized connection weight value is a binary value.

### Claim 6 (depends on 1)

The method of claim 1, wherein quantized connection weight value and said quantized activation value are binary values.

### Claim 7 (depends on 1)

The method of claim 1, wherein said quantized activation function and said quantized connection weight function are deterministic functions.

### Claim 8 (depends on 1)

The method of claim 1, wherein said quantized activation function and said quantized connection weight function are stochastic functions.

### Claim 9 (depends on 8)

The method of claim 8, wherein said stochastic functions are hard sigmoid functions.

### Claim 10 (depends on 1)

The method of claim 1, wherein said input is batch normalized.

### Claim 11 (depends on 1)

The method of claim 1, wherein said input is scaled and shifted according to at least one constant value.

### Claim 12 (depends on 1)

The method of claim 1, wherein said input is scaled and shifted according to a batch normalization calculated by approximating variance of values of at least one minibatch of said training set and normalizing said values of said at least one minibatch of said training set.

### Claim 13 (depends on 12)

The method of claim 12, wherein multiplications of said batch normalization are performed using left and right bit-shift operations.

### Claim 14 (depends on 2)

The method of claim 2, further comprising updating said plurality of floating-point connection weight values during said backpropagation sub-processes by performing multiplication operations using left and right bit-shift operations.

### Claim 15

A system for training neural networks, comprising:
a storage comprising a neural network model having a plurality of neurons each associated with a quantized activation function adapted to output a quantized activation value selected from a first finite set, said plurality of neurons are arranged in a plurality of layers and being connected by a plurality of connections each associated with a quantized connection weight function adapted to output a quantized connection weight value selected from a second finite set;
at least one processor coupled to the storage for executing a code comprising:
using a training set dataset to train said neural network model according to respective said quantized connection weight values, said training includes computing a plurality of weight gradients for backpropagation sub-processes by:
computing a plurality of neuron gradients, each said neuron gradient is of an output of a respective said quantized activation function in one layer of said plurality of layers with respect to an input of said respective quantized activation function and is calculated such that when an absolute value of said input is smaller than a positive constant threshold value, said respective neuron gradient is set as a positive constant value and when the absolute value of said input is smaller than said positive constant threshold value said neuron gradient is set to zero, and
updating a plurality of floating point connection weight values according to said plurality of weight gradients; and
outputting a trained quantized neural network formed as an outcome of said training process.

### Claim 16

A method for using a trained neural network, comprising:
providing a neural network trained by a training set and having a plurality of neurons each associated with a quantized activation function adapted to output a quantized activation value selected from a first finite set, said plurality of neurons are arranged in a plurality of layers and being connected by a plurality of connections each associated with a quantized connection weight value selected from a second finite set;
receiving new data element;
using said neural network trained for inferencing a conclusion regarding said data element by at least one forward pass process during which only bitwise operations are held to multiply respective said quantized connection weight values and respective said quantized activation values; and
outputting said conclusion.

### Claim 17 (depends on 16)

The method of claim 16, wherein said bitwise operations are XNOR-count operations and population count, operations.