Dataflow all-reduce for reconfigurable processor systems

A system for training parameters of a neural network using training data samples partitioned across a plurality of participating reconfigurable processors, each of the data samples including a plurality of input values and a set of at least one target output value, comprising:

• a plurality of N processing nodes, N>1, each k'th one of processing nodes in the plurality of N processing nodes including:
• a respective plurality of Mk first reconfigurable processors RPk.0 . . . RPk.Mk−1, each reconfigurable processor in the respective plurality of Mk first reconfigurable processors RPk.0 . . . RPk.Mk−1 being reconfigurable at a first level of configuration granularity,
• a respective k'th reconfigurable master network interface controller RUk, k=0, . . . , N−1, each master network interface controller in the respective k'th reconfigurable master network interface controller RUk, k=0, . . . , N−1 being reconfigurable at a second level of configuration granularity finer than the first level of configuration granularity, and
• a network interconnecting master network interface controllers in the respective k'th reconfigurable master network interface controller RUk, k=0, . . . , N−1;
• wherein each reconfigurable processor is configured by a host processor with one or more dataflow pipelines implementing a respective instance of a first dataflow segment, the first dataflow segment including, for a respective current subset of training data samples and a current set of neural network parameters, pipeline stages that:
• evaluate a neural network using input values of the respective current subset of training data samples and parameter values of the current set of neural network parameters, to calculate a set of at least one predicted output value,
• calculate a loss in dependence upon the predicted output value and a target output value in a set of at least one target output value,
• calculate a first intermediate result in dependence upon a parameter gradient with respect to each of at least a subset of neural network parameters in the current set of neural network parameters, and
• forward the first intermediate result to the respective k'th reconfigurable master network interface controller RUk without passing through the host processor;
• wherein each of the master network interface controllers is configured by the host processor with one or more dataflow pipelines implementing a respective instance of a second dataflow segment, the second dataflow segment including pipeline stages that:
• participate in an aggregation of first intermediate results from the first reconfigurable processors RPk.0 . . . RPk.Mk−1, to calculate a second intermediate result in dependence upon the first intermediate results,
• communicate via a peer-to-peer (P2P) protocol layer over the network and without passing through the host processor, to participate in an aggregation of second intermediate results from a second reconfigurable units RUk, k=0, . . . , N−1, to calculate a third intermediate result in dependence upon the second intermediate results, the third intermediate result indicative of an update of the current set of neural network parameters, and
• forward the third intermediate result to the first reconfigurable processors RPk.i, i=0, . . . , Mk−1, without passing through the host processor, for use in a subsequent evaluation of the neural network using a subsequent subset of training data samples;
• and
• wherein each of the reconfigurable master network interface controllers RUk is configured by the host processor with a respective local gradient buffer having N gradient buffer segments,
• wherein the pipeline stages in the second dataflow segment configured into each of the reconfigurable master network interface controllers RUk, for communicating over the network to participate in the aggregation of the second intermediate results from the second reconfigurable units RUk, k=0, . . . , N−1, includes pipeline stages implementing an accumulation phase of the aggregation followed by a distribution phase of the aggregation.