System with hybrid communication strategy for large-scale distributed deep learning

A distributed computing system comprising a computer comprising:

• a graphics processing unit (GPU) memory;
• a central processing unit (CPU) memory comprising a Key-Value Store (KVS) module;
• an execution engine module configured to run a deep learning (DL) program to create a plurality of operator graph layers in the graphics processing unit memory;
• a client library module configured to create a GPU-CPU synchronization (GCS) module for each of the plurality of operator graph layers;
• a coordination service module configured to compute network cost of a first and a second communication scheme and select, based on the network cost, one of the first and second communication scheme for transmitting data associated with one of the plurality of operator graph layers from a corresponding GCS module;
• and wherein the client library module is further configured to initiate a data transfer from the GCS module using the selected communication scheme, wherein the first communication scheme comprises broadcasting data associated with the one of the plurality of operator graph layers from the corresponding GCS module to one or more GCS modules directly, wherein the network cost associated with the first communication scheme can be computed as P2B(M+N), wherein P is a number of computers in the distributed system, B is a batch size, M and N are dimensions of a matrix associated with the operator graph layer.