Efficient data delivery for artificial intelligence systems in distributed storage environments

An apparatus comprising:

• a memory;
• a processing device, operatively coupled to the memory, configured to:
• identify, based on one or more machine learning models to be executed on a compute cluster comprising a plurality of graphics processing unit (GPU) servers, one or more transformations to apply to a dataset;
• generate multiple transformed datasets having different levels of precision or format;
• store the multiple transformed datasets;
• receive a request to transmit a transformed dataset to at least one GPU server of the plurality of GPU servers;
• select one of the multiple transformed datasets for transmission based on a resource availability of the at least one GPU server;
• and
• responsive to the request, transmit, to the at least one GPU server without re-performing the one or more transformations on the dataset, the selected one of the multiple transformed datasets.

A non-transitory computer readable storage medium storing instructions that, when executed, cause a processing device to:

• identify, based on one or more machine learning models to be executed on a compute cluster comprising a plurality of graphics processing unit (GPU) servers, one or more transformations to apply to a dataset;
• generate multiple transformed datasets having different levels of precision or format;
• store the multiple transformed datasets;
• receive a request to transmit a transformed dataset to at least one GPU server of the plurality of GPU servers;
• select one of the multiple transformed datasets for transmission based on a resource availability of the at least one GPU server;
• and
• responsive to the request, transmit, to the at least one GPU server without re-performing the one or more transformations on the dataset, the selected one of the multiple transformed datasets.