Systems and methods for intelligent data shuffling for high-performance distributed machine learning training

A training node for performing data shuffling for distributed machine learning training using a training data set, the training node comprising:

• a processor;
• and
• a memory unit operatively connected to the processor, the memory unit including instructions that when executed by the processor, cause the training node to perform the operations of:
• receiving one of N shards of training data, wherein the training data set is divided into a plurality of N shards comprising a plurality of data items, each of the respective N shards is sent to its respective training node in a distributed training network, and each of the plurality of data items is assigned to one of a first working set and a second working set such that each of the first and second working sets includes data items from a plurality of shards;
• performing training using the data items of the first working set that are the shard received by the node;
• upon completion of the training using the data items of the first working set, the node performing training using the data items of the second working set that are in the node's shard;
• and
• while the training node is performing training on its subset of shards of the second working set, randomly shuffling data items in the first working set across other nodes in the distributed training network to create a shuffled first working set.

A distributed training network for performing data shuffling for distributed machine learning training using a training data set, the distributed training network comprising:

• a plurality of training nodes, each training node of the plurality of training nodes comprising a processor and a memory unit operatively connected to the processor, the memory units of the plurality of nodes including instructions that when executed, cause the training nodes to perform the operations of:
• each of N training nodes of the plurality of training nodes receiving one of N shards of training data, wherein the training data set is divided into a plurality of N shards comprising a plurality of data items and each of the plurality of data items is assigned to one of a first working set and a second working set such that each of the first and second working sets includes data items from a plurality of shards;
• performing training using the data items of the first working set that are in each node's respective shard;
• upon completion of the training using the data items of the first working set, performing training using the data items of the second working set that are in each node's respective shard;
• and
• while the training nodes are performing training on their respective subsets of shards of the second working set, randomly shuffling data items in the first working set to create a shuffled first working set.