Training a neural network using a non-homogenous set of reconfigurable processors

A method for training parameters of a neural network using training data samples that include a plurality of input values and a target output value, the method utilizing a host system communicatively coupled to a processing node that includes a processor reconfigurable at a first level of configuration granularity and a controller reconfigurable at a second level of configuration granularity finer than the first level of configuration granularity, the controller communicatively coupled to the processor through an intra-node communication link, and to a network interface controller on a second processing node through a network, the method comprising:

• configuring the processor at the first level of configuration granularity by the host system to execute a first dataflow segment of the neural network having one or more dataflow pipelines;
• configuring the controller at the second level of configuration granularity by the host system to execute a second dataflow segment dependent upon a first intermediate result and a second intermediate result;
• generating, using the one or more dataflow pipelines, a predicted output value using a first subset of the plurality of input values and a set of neural network parameters;
• calculating, using the one or more dataflow pipelines, the first intermediate result for a parameter of the set of neural network parameters based on the predicted output value and the target output value;
• providing, by the processor, the first intermediate result to the controller;
• receiving respective second intermediate results, including the second intermediate result from the second processing node, from respective smart network interface controllers of a plurality of processing nodes;
• and
• generating, using a collective methodology at respective controllers, a third intermediate result by adding the first intermediate result and the second intermediate results, the third intermediate result indicative of an update of the parameter of the set of neural network parameters;
• wherein the respective smart network interface controllers of the plurality of processing nodes are connected in a ring configuration and the collective methodology comprises a uni-directional ring all-reduce;
• and
• wherein none of the first intermediate result, the second intermediate result, nor the third intermediate result, pass through the host system.

One or more non-transitory computer-readable media having one or more configuration files stored thereon, that when used by a host system to configure a processing node that includes a processor reconfigurable at a first level of configuration granularity and a controller reconfigurable at a second level of configuration granularity finer than the first level of configuration granularity, the controller communicatively coupled to the processor through an intra-node communication link, and to a network interface controller on a second processing node through a network, cause a method for training parameters of a neural network using training data samples that include a plurality of input values and a target output value to be performed, the method comprising:

• configuring the processor at the first level of configuration granularity by the host system to execute a first dataflow segment of the neural network having one or more dataflow pipelines;
• configuring the controller at the second level of configuration granularity by the host system to execute a second dataflow segment dependent upon a first intermediate result and a second intermediate result;
• generating, using the one or more dataflow pipelines, a predicted output value using a first subset of the plurality of input values and a set of neural network parameters;
• calculating, using the one or more dataflow pipelines, the first intermediate result for a parameter of the set of neural network parameters based on the predicted output value and the target output value;
• providing, by the processor, the first intermediate result to the controller;
• receiving respective second intermediate results, including the second intermediate result from the second processing node, from respective smart network interface controllers of a plurality of processing nodes;
• and
• generating, using a collective methodology at respective controllers, a third intermediate result by adding the first intermediate result and the second intermediate results, the third intermediate result indicative of an update of the parameter of the set of neural network parameters;
• wherein the respective smart network interface controllers of the plurality of processing nodes are connected in a ring configuration and the collective methodology comprises a uni-directional ring all-reduce;
• and
• wherein none of the first intermediate result, the second intermediate result, nor the third intermediate result, pass through the host system.