Accelerated neural network training using a pipelined resistive processing unit architecture

A method of performing neural network training, comprising:

• initializing a neural network hardware having a plurality of neurons, a plurality of arrays of weights, and a neuron control system, wherein the plurality of neurons comprises a layer of input neurons, one or more layers of hidden neurons, and a layer of output neurons, and each array of weights being configured to receive a plurality of discrete data points from a first layer of neurons and to produce a corresponding discrete data point to a second layer of neurons during a feed forward operation, each array of weights comprising a plurality of resistive processing units (RPU) having respective settable resistances;
• receiving, at each of the plurality of neurons, a sequence of discrete data points, one discrete data point during each of a plurality of operation cycles, wherein the discrete data point comprises an input current from an input of the neuron, and an output voltage pulse at an output of the neuron;
• sending, by the neuron control system, a control signal to each of the plurality of neurons based on a location of the neuron and a discrete data point associated with the neuron at the operation cycle to control an operation mode of the neuron, wherein the operation mode of the neuron comprises a feed forward mode, a back propagation mode and a weight update mode;
• performing computations at each of the plurality of neurons during the operation cycle to produce an output data point of the neuron;
• completing one iteration of neural network training when each of the sequence of discrete data points at each of the layer of input neurons has been processed through feed forward operation, back propagation operation, and weight update operation;
• and
• completing the neural network training when a predetermined termination condition is met.

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising:

• initializing a neural network hardware having a plurality of neurons, a plurality of arrays of weights, and a neuron control system, wherein the plurality of neurons comprises a layer of input neurons, one or more layers of hidden neurons, and a layer of output neurons, and each array of weights being configured to receive a plurality of discrete data points from a first layer of neurons and to produce a corresponding discrete data point to a second layer of neurons during a feed forward operation, each array of weights comprising a plurality of resistive processing units (RPU) having respective settable resistances;
• receiving, at each of the plurality of neurons, a sequence of discrete data points, one discrete data point during each of a plurality of operation cycles, wherein the discrete data point comprises an input current from an input of the neuron, and an output voltage pulse at an output of the neuron;
• sending, by the neuron control system, a control signal to each of the plurality of neurons based on a location of the neuron and a discrete data point associated with the neuron at the operation cycle to control an operation mode of the neuron, wherein the operation mode of the neuron comprises a feed forward mode, a back propagation mode and a weight update mode;
• performing computations at each of the plurality of neurons during the operation cycle to produce an output data point of the neuron;
• completing one iteration of neural network training when each of the sequence of discrete data points at each of the layer of input neurons has been processed through feed forward operation, back propagation operation, and weight update operation;
• and
• completing the neural network training when a predetermined termination condition is met.