Methods and systems for loading large datasets across cloud networks

A system comprising:

• one or more processors;
• and
• a non-transitory computer-readable medium storing instructions that when executed by the one or more processors cause the one or more processors to perform operations including:
• receiving a request to load a dataset into a first distributed processing network, wherein the dataset is located at a second distributed processing network;
• dividing the dataset into one or more data segments, wherein each data segment of the one or more data segments is associated with a data loader of one or more data loaders of the first distributed processing network;
• transferring the one or more data segments to a temporary location within a third distributed processing network, wherein the one or more data segments are transferred in response to receiving the request to load the dataset, and wherein the second distributed processing network is distinct from the third distributed processing network;
• detecting the one or more data segments at the temporary location by executing a call from the first distributed processing network;
• downloading, by one or more data loaders of the first distributed processing network, the one or more data segments from the temporary location to the first distributed processing network, wherein the download is initiated in response to detecting the one or more data segments at the temporary location, and wherein each data segment of the one or more data segments is downloaded by a separate data loader of the one or more data loaders;
• modifying a first data segment of the one or more data segments downloaded by the one or more data loaders in response to downloading the first data segment;
• and
• deleting the first data segment from the temporary location.

A non-transitory computer-readable medium storing instructions that when executed by one or more processors cause the one or more processors to perform operations including:

• receiving a request to load a dataset into a first distributed processing network, wherein the dataset is located at a second distributed processing network;
• dividing the dataset into one or more data segments, wherein each data segment of the one or more data segments is associated with a data loader of one or more data loaders of the first distributed processing network;
• transferring the one or more data segments to a temporary location within a third distributed processing network, wherein the one or more data segments are transferred in response to receiving the request to load the dataset, and wherein the second distributed processing network is distinct from the third distributed processing network;
• detecting the one or more data segments at the temporary location by executing a call from the first distributed processing network;
• downloading, by one or more data loaders of the first distributed processing network, the one or more data segments from the temporary location to the first distributed processing network, wherein the download is initiated in response to detecting the one or more data segments at the temporary location, and wherein each data segment of the one or more data segments is downloaded by a separate data loader of the one or more data loaders;
• modifying a first data segment of the one or more data segments downloaded by the one or more data loaders in response to downloading the first data segment;
• and
• deleting the first data segment from the temporary location.