Data prefetching for large data systems

A computer-implemented system for accelerating access to data in a storage system by an application, the system comprising:

• a detector module that detects the application accessing an initial file stored on the storage system;
• an application model database that stores an application model of the application, the application model comprising an access pattern of the application when accessing first files that have first metadata information similar to metadata information of the initial file;
• an access pattern predictor that identifies one or more second files stored in the storage system, the one or more second files having second metadata information that are similar to the first metadata information, and that filters the one or more second files by computing, for each second file, a probability of each second file to be accessed in a next time period, according to the access pattern of the application model, and comparing the probability to a threshold that incorporates a value for percentage of free cache;
• a prefetch module that prefetches from cold storage to cache memory those of the one or more second files for which the probability to be accessed exceeds the threshold;
• and
• a memory that stores the one or more second files that are prefetched;
• wherein the access pattern predictor associates a prediction confidence to each second file to be prefetched and the prefetch module prefetches each second file that is associated with a prediction confidence that exceeds a predetermined confidence threshold.

A computer program product for accelerating access to data in a storage system by an application, the computer program product comprising a computer readable non-transitory storage medium embodying program instructions, the program instructions being executable by a computerized system to cause:

• detecting the application accessing an initial file stored on the storage system;
• fetching, from an application model database, an application model of the application in response to detecting the access of the initial file, the application model comprising an access pattern of the application when accessing first files having first metadata information similar to metadata information of the initial file;
• identifying one or more second files stored in the storage system, the identified one or more second files having second metadata information that are similar to the first metadata information;
• filtering the one or more second files by computing, for each second file, a probability of each second file to be accessed in a next time period, according to the access pattern of the application model, and comparing the probability to a threshold that incorporates a value for percentage of free cache;
• and
• prefetching from cold storage to cache memory those of the one or more second files for which the probability to be accessed exceeds the threshold;
• wherein prefetching the identified one or more second files further comprises associating a prediction confidence to each second file to be prefetched and prefetching second files associated with a prediction confidence that exceeds a predetermined confidence threshold.