Machine learning pipeline performance acceleration with optimized data access interfaces using in-memory and distributed store

A system, comprising:

• an in-memory object store configured to store data objects and object handles;
• a persistent distributed storage configured to store data objects;
• and
• one or more processors communicatively coupled to the in-memory object store via a first one or more data access interfaces and communicatively coupled to the persistent distributed storage via a second one or more data access interfaces, the one or more processors configured to execute instructions stored in a memory to:
• identify a plurality of data access interfaces comprising at least a first data access interface for the persistent distributed storage and at least a second data access interface for the in-memory object store;
• issue a data operation request as part of a machine learning pipeline;
• obtain performance metrics for the plurality of data access interfaces;
• for a first instance of the data operation request, executing the data operation on the persistent storage using a best performing data access interface selected from the at least a second data access interfaces based on the performance metrics and a type of the data operation request, wherein executing the data operation request comprises writing data to the persistent storage, and storing, to the in-memory object store, an object handle pointing to the data written to the persistent storage;
• and
• for a second or later instance of the data operation, executing the data operation by accessing the object handle in the in-memory object store that directs the data operation to the data stored to the persistent storage.

A compute node, comprising:

• a processor;
• and
• a memory communicatively coupled to the processor and configured to store:
• a common store configuration (CSC) module executable, by the processor, to discover identify plurality of data access interfaces exposed by a first data store and a second data store and collect metrics corresponding to each of the plurality of data access interfaces;
• an optimized performance interface (OPI) module executable, by the processor, to:
• obtain, from the CSC module, the plurality of data access interfaces and map the plurality of data access interfaces to a plurality of wrapper interfaces, wherein a first wrapper interface corresponds to the first data store and the second interface corresponds to the second data store, and
• responsive to a data operation received by the OPI module, generate a recommendation request message requesting identification of a best performing data access interface from the plurality of data access interfaces;
• a resource performance analyzer (RPA) module executable, by the processor, to receive the recommendation request and return one or more of the best performing data access interfaces to the OPI module, wherein the OPI module performs the data operation using one of the one or more best performing data access;
• and
• zero-copy control engine executable, by the processor, to synchronize a plurality of parallel data operations by loading data into the first data store, and generating and storing object handles in the second data store directing the plurality of parallel data operations to the data loaded into the first data store.