Distributed deep learning system using a communication network for stochastic gradient descent calculations

A distributed deep learning system comprising:

• a plurality of learning nodes;
• and
• a plurality of computing interconnect devices connected to the plurality of learning nodes or other devices via a communication network;
• wherein each learning node of the plurality of learning nodes comprises:
• a gradient calculator that calculates a gradient of a loss function from a respective output result obtained by inputting learning data to a learning target neural network corresponding to the learning node;
• a first transmitter that converts a calculation result of the gradient calculator into a first packet and transmits the first packet to a computing interconnect device of the plurality of computing interconnect devices connected to the learning node;
• a first receiver that receives a second packet transmitted from the computing interconnect device connected to the learning node and acquires a value stored in the second packet;
• and
• a constituent-parameter updater that updates a constituent parameter of the learning target neural network based on the value stored in the second packet;
• wherein a first computing interconnect device of the plurality of computing interconnect devices that is positioned at highest order among the plurality of computing interconnect devices comprises:
• a second receiver that:
• receives a third packet transmitted from a second computing interconnect device of the plurality of computing interconnect devices, the second computing interconnect device is at an immediately lower order than the first computing interconnect device;
• and
• receives a fourth packet transmitted from a first learning node of the plurality of learning nodes that is connected to the first computing interconnect device and
• acquires a value of a gradient stored in the third packet and a value of a gradient stored in the fourth packet;
• a first calculator that receives and performs calculation processing on the value of the gradient stored in the third packet and the value of the gradient stored in the fourth packet;
• and
• a second transmitter that converts a calculation result of the first calculator into a fifth packet and transmits the fifth packet to the second computing interconnect device at the immediately lower order than the first computing interconnect device and to the first learning node connected to the first computing interconnect device, and
• wherein a another computing interconnect device of the plurality of computing interconnect devices at a lower order than the first computing interconnect device comprises:
• a third receiver that:
• receives a sixth packet transmitted from a third computing interconnect device at an immediately lower order than the another computing interconnect device or transmitted from a second learning node of the plurality of learning nodes connected to the another computing interconnect device;
• and
• acquires a value of a gradient stored in the sixth packet;
• a second calculator that receives and performs calculation processing on the value of the gradient in the sixth packet and performs calculation processing;
• and
• a third transmitter that converts a calculation result of the second calculator into a seventh packet and transmits the seventh packet to a fourth computing interconnect device at an immediately higher order than the another computing interconnect device.

A distributed deep learning system comprising:

• a plurality of learning nodes;
• and
• a plurality of computing interconnect devices connected to the plurality of learning nodes or other devices via a communication network;
• wherein the plurality of computing interconnect devices are connected by a ring-type communication network that performs communication in only one direction;
• wherein each learning node of the plurality of learning nodes includes:
• a gradient calculator that calculates a gradient of a loss function from a output result obtained by inputting learning data to a learning target neural network of the learning node;
• a first transmitter that converts a calculation result of the gradient calculator into a first packet and transmits the first packet to a computing interconnect device of the plurality of computing interconnect devices connected to the learning node;
• a first receiver that receives a second packet transmitted from the computing interconnect device connected to the learning node and acquires a value stored in the second packet;
• and
• a constituent-parameter updater that updates a constituent parameter of the learning target neural network based on the value stored in the second packet;
• wherein a first computing interconnect device of the plurality of computing interconnect devices comprises:
• a constituent parameter memory that stores a respective constituent parameter of a learning target neural network of each of the plurality of learning nodes;
• a second receiver that receives a third packet transmitted from a first learning node connected to the first computing interconnect device and acquires a value of a gradient stored in the third packet;
• a third receiver that receives a fourth packet transmitted from a second computing interconnect device adjacent and upstream to the first computing interconnect device and acquires a value of a gradient stored in the fourth packet;
• a first sorter that outputs the value of the gradient stored in the fourth packet and in response to acquiring a value of a gradient that was previously acquired discards the value of the gradient that was previously acquired;
• a constituent-parameter-update calculator that calculates, based on the value of the gradient output from the first sorter and a value of a first constituent parameter of a first learning target neural network stored in the constituent parameter memory, an updated value of the first constituent parameter and updates the value of the first constituent parameter stored in the constituent parameter memory, wherein the first learning target neural network corresponds to one of the plurality of learning nodes;
• a second transmitter that converts the value of the gradient stored in the third packet or the updated value of the first constituent parameter into a fifth packet and transmits the fifth packet to a third computing interconnect device adjacent and downstream to the first computing interconnect device;
• and
• a third transmitter that converts the updated value of the first constituent parameter into a sixth packet and transmits the sixth packet to the first learning node connected to the first computing interconnect device;
• and
• wherein a fourth computing interconnect device of the plurality of computing interconnect devices and different than the first computing interconnect device comprises:
• a fourth receiver that receives a seventh packet transmitted from a fifth computing interconnect device adjacent and upstream to the fourth computing interconnect device upstream and acquires a value stored in the seventh packet;
• a fifth receiver that receives an eighth packet transmitted from a second learning node connected to the fourth computing interconnect device and acquires a value of a gradient stored in the eighth packet;
• a second sorter that sorts the value stored in the seventh packet as a value of incomplete calculation processing for a gradient and the updated value of the first constituent parameter;
• a calculator that receives, as an input, the value of the incomplete calculation processing and the value of the gradient stored in the eighth packet and performs calculation processing;
• a fourth transmitter that converts a calculation result of the calculator or the updated value of the first constituent parameter into ninth a packet and transmits the ninth packet to a sixth computing interconnect device adjacent and downstream to the fourth computing interconnect device;
• and
• a fifth transmitter that converts the value after the updated value of the first constituent parameter into a tenth packet and transmits the tenth packet to the second learning node connected to the fourth computing interconnect device;
• and
• wherein a respective constituent-parameter updater of each of the plurality of learning nodes overwrites a respective constituent parameter of a respective learning target neural network with the updated value of the first constituent parameter.