Sequential file restore performance using dynamic prefetch based on file locality

A method of providing a dynamic prefetch process based on file locality in a deduplication system, comprising:

• tracking a locality of data elements in a batched data stream, wherein the locality per batch corresponds to a number of different container IDs (CIDs) accessed by the data elements;
• determining whether the locality within a batch exceeds a threshold separating acceptable versus non-acceptable distribution of data accesses by the data elements over a plurality of container IDs;
• processing the data elements in a serial prefetch process for a batch that does not exceed the threshold;
• dividing a batch into a plurality of smaller sub-batches for a batch that does exceed the threshold and executing the prefetch process in parallel for the sub-batches;
• and
• determining a load on the system based on at least one of:
• number of streams, number of background jobs, and I/O response time, wherein a numerically higher threshold value indicates closer locality among data elements based on a lower number of CIDs accessed, and a numerically lower threshold value indicates a higher degree of distribution of the data elements among a greater number of CIDs accessed, the method further comprising determining a number of smaller sub-batches dividing the batch based on a locality a first batch and the load.

A method comprising:

• batching a data stream into a plurality of batches, each having a number of data segments for prefetching by a prefetch operation;
• determining a locality of a first batch of the data stream based on a number of different container IDs (CIDs) accessed by data segments in the batch;
• determining a load on the system based on at least one of:
• number of streams, number of background jobs, and I/O response time;
• determining a performance of the prefetch operation for the batch based on the load and the locality;
• dividing the batch into a plurality of sub-batches if the locality is lower than a defined threshold;
• and
• performing the prefetch operation in parallel for each of the sub-batches, wherein a numerically higher threshold value indicates closer locality among data elements based on a lower number of CIDs accessed, and a numerically lower threshold value indicates a higher degree of distribution of the data elements among a greater number of CIDs accessed, the method further comprising determining a number of smaller sub-batches dividing the batch based on a locality a first batch and the load.