Neural Network Training Method and Related Device
AssigneeHuawei Technologies Co., Ltd.
InventorsYixing Xu, Kai Han, Yehui Tang, Yunhe Wang, Chunjing Xu
Life of this patent
Estimating the term…
Abstract
A neural network training method includes performing, in a forward propagation process, binarization processing on a target weight by using a binarization function, and using data obtained through the binarization processing as a weight of a first neural network layer in a neural network; and calculating, in a backward propagation process, a gradient of a loss function with respect to the target weight by using a gradient of a fitting function as a gradient of the binarization function.
Claims20
A neural network training method, comprising:
- performing, in a forward propagation process and using a binarization function, binarization processing on a target weight to obtain a weight of a first neural network layer in a neural network, or on an activation value of a second neural network layer in neural network to obtain an input of the first neural network layer;
- determining a fitting function based on series expansion of the binarization function;
- and
- calculating, in a backward propagation process, a first gradient of a loss function with respect to the target weight using a second gradient of the fitting function as a third gradient of the binarization function.
A training device, comprising:
- a memory configured to store instructions;
- and
- one or more processors coupled to the memory and configured to:
- perform, in a forward propagation process and using a binarization function, binarization processing on a target weight to obtain a weight of a first neural network layer in a neural network, or on an activation value of a second neural network layer in the neural network to obtain an input of the first neural network layer;
- determine a fitting function based on series expansion of the binarization function;
- and
- calculate, in a backward propagation process, a first gradient of a loss function with respect to the target weight by using a second gradient of a fitting function as a third gradient of the binarization function.
A computer program product comprising computer-executable instructions that are stored on a non-transitory computer-readable storage medium and that, when executed by a processor, cause a training device to:
- perform, in a forward propagation process and using a binarization function, binarization processing on a target weight to obtain a weight of a first neural network layer in a neural network, or on an activation value of a second neural network layer in the neural network to obtain an input of the first neural network layer;
- determine a fitting function based on series expansion of binarization function;
- and
- calculate, in a backward propagation process, a first gradient of a loss function with respect to the target weight using a second gradient of a fitting function as a third gradient of the binarization function.
Dependencies are inferred from the claim text.