Distributed parallel computation with acceleration devices

A non-transitory computer readable storage medium having computer readable instructions stored thereon that, when executed by a computer, implement a method for distributed computing between a host computer and at least one accelerator device interconnected through a network, the method comprising:

• profiling a data transfer rate and a computation rate for a range of data sizes to find an optimal chunk size for the data transfer through the network;
• splitting or aggregating a size of a data stored in a memory in the host computer for encapsulating the data into a chunk with the optimal chunk size;
• dispatching the encapsulated data to the accelerator device;
• and
• instructing pipeline computation to the accelerator device with respect to the encapsulated data received.

A host computer for distributed computing between the host computer and at least one accelerator device interconnected through a network, the host computer comprising:

• a profiler part configured to profile a data transfer rate and a computation rate for a range of data sizes;
• a size optimizer part configured to find an optimal chunk size for the data transfer through the network from the profiled data transfer rate and the computation rate;
• an encapsulation part configured to encapsulate a data stored in a memory of the host computer by splitting or aggregating into a chunk with the optimal chunk size;
• and
• a dispatch part configured to dispatch the encapsulated data to the accelerator device and instructing pipeline computation to the accelerator device with respect to the encapsulated data received.