Statistical classifier with reduced weight memory requirements
AssigneeApple Computer, Inc.
InventorRichard F. Lyon
Life of this patent
Estimating the term…
Abstract
A neural network has reduced requirements for storing intermodal weight values, as a result of a dual-precision training process. In the forward propagation of training samples, low-resolution weight values are employed. During back-propagation of errors to train the network, higher-resolution values are used. After training, only the lower resolution values need to be stored for further run-time operation, thereby reducing memory requirements.
Claims13
A method for training and operating a neural network of the type having plural layers of nodes where the nodes of one layer are connected to the nodes of a succeeding layer, and each connection has a weight value associated with it, comprising the steps of:
- storing a first set of digital weight values that are respectively associated with said connections, each of the weight values in said first set having a first predetermined number of bits;
- storing a second set of digital weight values that are respectively associated with said connections, each of the weight values in said second set having a second predetermined number of bits;
- supplying a sample training pattern to the neural network and generating a set of node activation values including output values in accordance with said first set of digital weight values;
- determining a set of error values based on the difference between the output values and a set of target values;
- computing a set of weight changes dependent on the error values, the node activation values, and the current values of at least said first set of weight values, said weight changes having a greater number of bits of precision than said first number of bits;
- adding said weight changes to a set of digital values formed by concatenating said first set of weight values as high-order parts and said second set of weight values as corresponding low-order fraction parts, to produce a set of high-precision updated weights;
- substituting said first number of bits of the high-order bits of said high-precision updated weights for said first set of digital weight values;
- and
- substituting said second number of bits of the subsequent bits of said high-precision updated weights for said second set of digital weight values.
A method for training and operating a neural network of the type having plural layers of nodes where the nodes of one layer are connected to the nodes of a succeeding layer, and each connection has a weight value associated with it, comprising the steps of:
- storing a first set of digital weight values that are respectively associated with said connections, each of the weight values in said first set having a first predetermined number of bits which defines a first resolution;
- providing a sample training pattern to the neural network and generating an output value in accordance with said first set of digital weight values;
- determining an error value based on the difference between the output value and a target value;
- calculating a weight adjustment value based on said error value, said weight adjustment value having a greater number of bits than said first predetermined number, to thereby define a second resolution which is higher than said first resolution;
- and
- back-propagating said weight adjustment value through the neural network and modifying said stored first set of weight values in accordance therewith.
Dependencies are inferred from the claim text.