System for efficient large-scale data distribution in distributed and parallel processing environment

A method for efficient large-scale data distribution in a distributed and parallel processing environment comprising:

• executing a plurality of processes by a set of interconnected processors,
• wherein at each of the processes on each of the interconnected processors receives input data defining:
• a total number (P) of the interconnected processors;
• an identifier (g) identifying the interconnected processor where the process is executed thereon;
• a set (G) of sparsified gradients data at the interconnected processor;
• and
• a total number (k) of non-zero elements in the set (G) of sparsified gradients data;
• wherein each of the processes further comprises:
• initializing a set (mask) of zero data of the same dimension as the set (G) of sparsified gradients data;
• extracting the non-zero elements in the set of sparsified gradients data into a first data array and the indices of the non-zero elements in the set of sparsified gradients into a second data array (I);
• appending the second data array to the end of the first data array to form a data array (sends);
• setting the zero data in the set (mask) of zero data at the indices of non-zero elements in the set of sparsified gradients to 1;
• initializing a data array (recvs) of the same dimension as the data array (sends) to receive data from one other processor in the plurality of interconnected processors;
• initializing a data array (peerMasks) of size P;
• and
• initializing each of the processes on each of the interconnected processors to perform nRounds times, wherein nRounds equals to log2 P, for each iteration of each of the processes on each of the interconnected processors until nRounds rounds of iterations;
• wherein the method further comprises:
• choosing a first processor and a second processor of the interconnected processors to exchange data with each other, wherein the first processor is peerDistance away from the second processor, and the peerDistance is 2i-1 away with i increases at each iteration from 1 to nRounds;
• and
• processing input information comprising at least exchanging data between the first and second processors by the plurality of the interconnected processors collectively, so as to generate and output result.