Systems and methods for auto-tiered data storage for data intensive applications

A server for training a machine learning model based on a training dataset, the training dataset being formed by data objects distributed across a virtual object storage service, the server being configured to access the virtual object storage service, the server comprising a processor and a memory configured to store instructions which, upon being executed by the processor, cause the server to:

• fetch, from the virtual object storage service, the training dataset;
• copy the fetched training dataset on a first local storage device, the first local storage device being communicably connected to the server;
• maintain a list of modifications executed on the training dataset that occurred on the virtual object storage service since the fetched training dataset has been stored on the first local storage device, each entry of the list of modifications comprising an identifier of a data object of the training dataset, a type of modification made to the data object and a temporal indication associated with a modification made to the data object;
• after receiving a request to initiate training of the machine learning model, generate a synchronized training dataset by applying the list of modifications to the fetched training dataset, the synchronized training dataset mirroring the training dataset stored in the virtual object storage service;
• store the synchronized training dataset in a second local storage device, the second local storage device being communicably connected to the server, the second local storage device having a lower data retrieval latency than the first local storage device;
• and
• fetch training data from the synchronized training dataset stored in the second local storage device as the training of the machine learning model is executed.

A method for providing a user device with access to a dataset, the dataset being formed by data objects distributed across a virtual object storage service, the method being executable by a server communicably connected to the user device and configured to access the virtual object storage service, the method comprising:

• fetching, by the server, from the virtual object storage service, the dataset;
• copying the fetched dataset on a first local storage device, the first local storage device being communicably connected to the server;
• maintaining, by the server, a list of modifications executed on the dataset distributed across the virtual object storage service that occurred since the fetched dataset has been stored on the first local storage device, each entry of the list of modifications comprising an identifier of a data object of the dataset, a type of modification made to the data object and a temporal indication associated with a modification made to the data object;
• after receiving, by the server from the user device, a request to access the dataset, generating a synchronized dataset by applying the list of modifications to the fetched dataset, the synchronized dataset mirroring the dataset stored in the virtual object storage service;
• storing the synchronized dataset in a second local storage device;
• and
• fetching data from the synchronized dataset stored in the second local storage device as the access to the dataset is provided to the user device.