DNN Training with Asymmetric RPU Devices

A method of training a DNN, the method comprising the steps of:

• providing a weight matrix (W) as a linear combination of a matrix A and a matrix C, wherein the matrix A and the matrix C are formed by stored conductance values in separate cross-point arrays A and C of RPU devices, each having conductive row wires orthogonal to conductive column wires, and the RPU devices separating intersections between the conductive row wires and the conductive column wires, whereby the RPU devices function as weighted connections between neurons in the DNN;
• applying repeated voltage pulses to the RPU devices in the cross-point array A until all of the RPU devices in the cross-point array A converge to their own symmetry point;
• copying zero-weight conductance values for the RPU devices in the cross-point array A to a reference array;
• transmitting, in a forward cycle, an input vector x as voltage pulses through the conductive column wires of the cross-point array A and the cross-point array C and reading a resulting output vector y as current output from the conductive row wires of the cross-point array A and the cross-point array C;
• transmitting, in a backward cycle, an error signal δ calculated from the output vector y as voltage pulses through the conductive row wires of the cross-point array A and the cross-point array C and reading a resulting output vector z as current output from the conductive column wires of the cross-point array A and the cross-point array C;
• transmitting, in an update of the cross-point array A, the input vector x as voltage pulses through the conductive column wires of the cross-point array A simultaneously with the error signal δ as voltage pulses through the conductive row wires of the cross-point array A;
• transmitting, in a forward cycle, an input vector ei as voltage pulses through the conductive column wires of the cross-point array A and reading a resulting output vector y′ as current output from the conductive row wires of the cross-point array A;
• calculating ƒ(y′) using y′;
• and
• transmitting, in an update of the cross-point array C, the input vector ei as voltage pulses through the conductive column wires of the cross-point array C simultaneously with ƒ(y′) as voltage pulses through the conductive row wires of the cross-point array C, wherein ƒ(y′)=y′ only if |y′|>a threshold T, otherwise ƒ(y′)=0.

A method of training a DNN, the method comprising the steps of:

• providing a weight matrix (W) as a linear combination of a matrix A and a matrix C, wherein the matrix A and the matrix C are formed by stored conductance values in separate cross-point arrays A and C of RPU devices, each having conductive row wires orthogonal to conductive column wires, and the RPU devices separating intersections between the conductive row wires and the conductive column wires, whereby the RPU devices function as weighted connections between neurons in the DNN;
• applying repeated voltage pulses to the RPU devices in the cross-point array A until all of the RPU devices in the cross-point array A converge to their own symmetry point;
• copying zero-weight conductance values for the RPU devices in the cross-point array A to a reference array;
• transmitting, in a forward cycle, an input vector x as voltage pulses through the conductive column wires of the cross-point array A and the cross-point array C and reading a resulting output vector y as current output from the conductive row wires of the cross-point array A and the cross-point array C;
• calculating a linear combination of the output vector y from the cross-point array A and the cross-point array C;
• transmitting, in a backward cycle, an error signal S calculated from the output vector y as voltage pulses through the conductive row wires of the cross-point array A and the cross-point array C and reading a resulting output vector z as current output from the conductive column wires of the cross-point array A and the cross-point array C;
• calculating a linear combination of the output vector z from the cross-point array A and the cross-point array C;
• transmitting, in an update of the cross-point array A, the input vector x as voltage pulses through the conductive column wires of the cross-point array A simultaneously with the error signal δ as voltage pulses through the conductive row wires of the cross-point array A;
• transmitting, in a forward cycle, an input vector ei as voltage pulses through the conductive column wires of the cross-point array A and reading a resulting output vector y′ as current output from the conductive row wires of the cross-point array A;
• calculating ƒ(y′) using y′;
• transmitting, in an update of the cross-point array C, the input vector ei as voltage pulses through the conductive column wires of the cross-point array C simultaneously with ƒ(y′) as voltage pulses through the conductive row wires of the cross-point array C, wherein ƒ(y′)=y′ only if |y′|>a threshold T, otherwise ƒ(y′)=0;
• and
• repeating the steps of the method until a convergence criterion is met.