Task synchronization for accelerated deep learning

An apparatus comprising:

• means for sending a first fabric packet by a sending processing element, the first fabric packet comprising a virtual channel specifier specifying one of a plurality of virtual channels;
• means for routing the first fabric packet from the sending processing element to a receiving processing element via one or more of routing elements in accordance with the virtual channel specifier;
• in the receiving processing element, means for picking for processing the first fabric packet, the picking in accordance with a respective block/unblock state comprised in the receiving processing element and maintained for each of the plurality of virtual channels, the picking excluding the first fabric packet from being picked until at least the respective block/unblock state maintained for the specified virtual channel is in the unblock state;
• wherein the sending processing element implements at least a portion of a first node of a plurality of nodes of a dataflow graph and the receiving processing element implements at least a portion of a second node of the dataflow graph, and the sending processing element and the receiving processing element are respective instances of a plurality of processing elements interconnected as a fabric;
• and
• wherein the first fabric packet is an instance of a plurality of fabric packets and the means for routing of the first fabric packet is a part of means for routing between the processing elements of the fabric, and wherein the means for routing between the processing elements of the fabric comprises means for each processing element transmitting each of one or more of the plurality of fabric packets over a selected group of preconfigured groups of one or more physical couplings between connected neighbors of the processing element, and the virtual channel specifier of each transmitted fabric packet is used to select the group.

A method comprising:

• specifying communications and computations respectively corresponding to a plurality of nodes of a dataflow graph, wherein a sending processing element implements at least a portion of a first node of the dataflow graph and a receiving processing element implements at least a portion of a second node of the dataflow graph, and the sending processing element and the receiving processing element are respective instances of a plurality of processing elements interconnected as a fabric;
• initializing the fabric with all parameters and task software required for a concurrent execution of the communications and computations respectively corresponding to the dataflow graph;
• sending a first fabric packet by the sending processing element, the first fabric packet comprising a virtual channel specifier specifying one of a plurality of virtual channels;
• routing the first fabric packet from the sending processing element to the receiving processing element via one or more of routing elements in accordance with the virtual channel specifier;
• in the receiving processing element, the virtual channel specifier of the first fabric packet identifying a respective one of a plurality of queues of the receiving processing element, the respective queue holding at least a task specifier of the first fabric packet;
• and
• in the receiving processing element, picking for processing the first fabric packet, the picking in accordance with a respective block/unblock state maintained for each of the plurality of queues of the receiving processing element, the picking excluding the first fabric packet from being picked until at least the respective block/unblock state maintained for the respective queue is in the unblock state.