Distributed computing architecture for large model deep learning

A system comprising:

• a processor;
• and
• a computer-readable storage medium storing program instructions for deep learning model training which, when executed by the processor, are configured to cause the processor to perform a method comprising:
• generating a model mapping table (MMT) storing information regarding respective portions of a deep learning model distributed amongst a plurality of interconnected host nodes, wherein respective host nodes comprise at least one central processing unit (CPU), at least one CPU memory, at least one graphics processing unit (GPU), and at least one GPU memory, wherein the deep learning model comprises an amount of data larger than an amount of memory in any respective host node of the plurality of interconnected host nodes;
• and
• training the deep learning model by training the respective portions of the deep learning model on the plurality of interconnected host nodes, the training comprising:
• receiving a request from a requesting GPU for a first portion of the deep learning model, wherein the requesting GPU is associated with a requesting GPU memory and a requesting host node;
• identifying a first host node of the plurality of interconnected host nodes storing the first portion of the deep learning model based on information in the MMT;
• transferring the first portion of the deep learning model from the first host node to the requesting host node;
• providing a first copy of the first portion of the deep learning model from the requesting host node to the requesting GPU memory;
• performing processing, by the requesting GPU, on the first copy of the first portion of the deep learning model stored in the requesting GPU memory;
• synchronizing the first copy of the first portion of the deep learning model with the first portion of the deep learning model in response to performing processing;
• and
• updating the MMT based on synchronizing the first copy of the first portion of the deep learning model.

A computer program product comprising a computer readable storage medium, wherein the computer readable storage medium does not comprise a transitory signal per se, wherein the computer readable storage medium stores instructions executable by a processor to cause the processor to perform a method comprising:

• generating a model mapping table (MMT) storing information regarding respective portions of a deep learning model distributed amongst a plurality of interconnected host nodes, wherein respective host nodes comprise at least one central processing unit (CPU), at least one CPU memory, at least one graphics processing unit (GPU), and at least one GPU memory, wherein the deep learning model comprises an amount of data larger than an amount of memory in any respective host node of the plurality of interconnected host nodes;
• and
• outputting a trained deep learning model by training the respective portions of the deep learning model on the plurality of interconnected host nodes, wherein training respective portions of the deep learning model comprises transferring, using a message passing interface (MPI) remote memory access (RMA) protocol, respective portions of the deep learning model between respective host nodes of the plurality of interconnected host nodes and providing respective copies of the respective portions of the deep learning model to respective GPU memories for processing by respective GPUs.