Methods, systems, and computer readable media for providing and using shuffle templates to distribute data among workers comprising compute resources in a data center

A system for providing shuffle templates and using the shuffle templates to implement shuffling of data among workers comprising compute resources in a data center, the system comprising:

• a shuffle manager configured for:
• providing an application programming interface (API) through which applications can select shuffle templates and specify data to be processed by the workers in the data center using the shuffle templates to distribute the data as messages transmitted among the workers;
• receiving, via the API, a call for a shuffle template, the call including a shuffle template identifier for one of the shuffle templates and source and destination identifiers respectively identifying sources and destinations of data to be processed by the workers;
• selecting the shuffle template identified by the call for the shuffle template;
• and
• providing the shuffle template to the workers;
• and
• workers configured for using the shuffle template to generate a shuffle plan and using the shuffle plan to shuffle the messages among the workers between the sources and the destinations.

A non-transitory computer readable medium having stored thereon executable instructions that when executed by at least one processor of at least one computer cause the at least one computer to perform steps comprising:

• providing an application programming interface (API) through which applications can select shuffle templates and specify data to be processed by workers in a data center using the shuffle templates to distribute the data as messages transmitted among the workers;
• receiving, via the API, a call for a shuffle template, the call including a shuffle template identifier for one of the shuffle templates and source and destination identifiers respectively identifying sources and destinations of data to be processed by the workers;
• selecting the shuffle template identified by the call for the shuffle template;
• providing the shuffle template to the workers;
• and
• using the shuffle template to generate a shuffle plan and using the shuffle plan to shuffle the messages among the workers between the sources and the destinations.