US 10,592,149 B1 · Dynamic de-duplication methodologies for efficient resource utilization on de-duplication system

Abstract

Embodiments are described for reducing memory usage and disk I/O's for reading and writing streams in a deduplication storage system. Generations of backups of a stream can be tracked for the stream. In the first generation (first backup), stream data is ingested, segmented, and written to disk without looking up a fingerprint of each segment, to save disk I/O's and memory. After the first generation, an amount of stream data processed is tracked, and at dynamically tunable increments of data processed, a fingerprint index lookup is performed to disk. The fingerprint index is read into memory and subsequent fingerprint lookups are preformed to cache, until a next dynamically tunable fingerprint lookup to disk. The dynamically tunable increment of processed data can be randomly reset.

Claims24

Dependent claims

A computer-implemented method comprising:

determining that a previous generation of stream of data has been deduplicated and written to a storage device as a plurality of segments of data, each segment having a fingerprint;
determining a hit ratio of fingerprint lookups in a fingerprint index cache of the segment storage device, over at least a subset of the plurality of segments of data of the previous generation of the stream of data;
for a subsequent generation of the stream of data being written to the storage device, determining a first fingerprint index lookup threshold that specifies a first amount of data that will be read from the subsequent stream, after which a fingerprint index lookup to the segment storage device will be performed, the index lookup threshold based at least in part on the hit ratio of the segment fingerprint lookups in the fingerprint index cache of the previous generation of the stream.

A non-transitory computer-readable medium programmed with executable instructions that, when executed by a processing system having at least one hardware processor, perform operations comprising:

determining that a previous generation of stream of data has been deduplicated and written to a storage device as a plurality of segments of data, each segment having a fingerprint;
determining a hit ratio of fingerprint lookups in a fingerprint index cache of the segment storage device, over at least a subset of the plurality of segments of data of the previous generation of the stream of data;
for a subsequent generation of the stream of data being written to the storage device, determining a first fingerprint index lookup threshold that specifies a first amount of data that will be read from the subsequent stream, after which a fingerprint index lookup to the segment storage device will be performed, the index lookup threshold based at least in part on the hit ratio of the segment fingerprint lookups in the fingerprint index cache of the previous generation of the stream.

A system comprising:

a processing system having at least one hardware processor, the processing system coupled to a memory programmed with executable instructions that, when executed by the processing system, perform operations comprising:
determining that a previous generation of stream of data has been deduplicated and written to a storage device as a plurality of segments of data, each segment having a fingerprint;
determining a hit ratio of fingerprint lookups in a fingerprint index cache of a the segment storage device, over at least a subset of the plurality of segments of data of the previous generation of the stream of data;
for a subsequent generation of the stream of data being written to the storage device, determining a first fingerprint index lookup threshold that specifies a first amount of data that will be read from the subsequent stream, after which a fingerprint index lookup to the segment storage device will be performed, the index lookup threshold based at least in part on the hit ratio of the segment fingerprint lookups in the fingerprint index cache of the previous generation of the stream.

1 more element

Dependencies are inferred from the claim text.

Dynamic de-duplication methodologies for efficient resource utilization on de-duplication system

Life of this patent

Dynamic de-duplication methodologies for efficient resource utilization on de-duplication system

Life of this patent

Abstract

Claims24