Centralized architecture for distributed data parallel training

A method comprising:

• determining, by a centralized process in a distributed data parallel training environment used to train a model via data parallelism, a respective state of each training worker process from a plurality of training worker processes in the distributed data parallel training environment, the model comprising an artificial intelligence (AI) or machine learning (ML) model;
• determining, by the centralized process based on the respective state of each training worker process, a respective task that one or more training worker processes of the plurality of training worker processes should perform with respect to at least one of a local replica of the model and training data associated with the local replica of the model;
• and
• sending, by the centralized process to the one or more training worker processes, one or more instructions to perform the respective task with respect to at least one of the local replica of the model and the training data.

A non-transitory computer-readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to:

• determine, by a centralized process in a distributed data parallel training environment used to train a model via data parallelism, a respective state of each training worker process from a plurality of training worker processes in the distributed data parallel training environment, the model comprising an artificial intelligence (AI) or machine learning (ML) model;
• determine, by the centralized process based on the respective state of each training worker process, a respective task that one or more training worker processes of the plurality of training worker processes should perform with respect to at least one of a local replica of the model and training data associated with the local replica of the model;
• and
• send, by the centralized process to the one or more training worker processes, one or more instructions to perform the respective task with respect to at least one of the local replica of the model and the training data.