Method or an apparatus for having perfect deduplication, adapted for saving space in a deduplication file system
AssigneeEMC IP Holding Company LLC
InventorsRamprasad Chinthekindi, Nitin Madan, Abhinav Duggal, Lan Bai
Life of this patent
Estimating the term…
Abstract
A data management device includes a persistent storage and a processor. The persistent storage includes an object storage. The processor generates a collision free hash function based on segments stored in the object storage. The processor generates a hash vector using the collision free hash function. The processor deduplicates the segments using the hash vector. The processor stores the deduplicated segments in the object storage.
Claims11
A data management device, comprising:
- a persistent storage comprising an object storage;
- and
- a processor programmed to perform a method, comprising:
- making a first determination that segments stored in the object storage are cold;
- generating, based on the first determination, a hash function based on the segments;
- generating a hash vector using the hash function, wherein the hash vector is a perfect hash live vector comprising a number of bits corresponding to a number of unique segments stored in the object storage;
- deduplicating the segments using the hash vector, wherein deduplicating the segment comprises:
- matching each fingerprint of each segment to a plurality of bits of the hash vector;
- making a second determination that a plurality of fingerprints match a first bit of the plurality of bits;
- identifying, based on the second determination, a first segment associated with a first fingerprint of the plurality of fingerprints;
- marking the first segment as unique;
- and
- deleting all of the segments associated with the plurality of fingerprints other than the first segment;
- and
- storing the deduplicated segments in the object storage.
A method of operating a data management device, comprising:
- making a first determination, by the data management device, that segments stored in an object storage are cold;
- generating, based on the first determination, a hash function based on the segments;
- generating a hash vector using the hash function, wherein the hash vector is a perfect hash live vector comprising a number of bits corresponding to a number of unique segments stored in the object storage;
- deduplicating the segments using the hash vector, wherein deduplicating the segment comprises:
- matching each fingerprint of each segment to a plurality of bits of the hash vector:
- making a second determination that a plurality of fingerprints match a first bit of the plurality of bits;
- identifying, based on the second determination, a first segment associated with a first fingerprint of the plurality of fingerprints;
- marking the first segment as unique;
- and
- deleting all of the segments associated with the plurality of fingerprints other than the first segment;
- and
- storing the deduplicated segments in the object storage.
A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for operating a data management device, the method comprising:
- making a first determination, by the data management device, that segments stored in an object storage are cold;
- generating, based on the first determination, a hash function based on the segments;
- generating a hash vector using the hash function, wherein the hash vector is a perfect hash live vector comprising a number of bits corresponding to a number of unique segments stored in the object storage;
- deduplicating the segments using the hash vector, wherein deduplicating the segment comprises:
- matching each fingerprint of each segment to a plurality of bits of the hash vector;
- making a second determination that a plurality of fingerprints match a first bit of the plurality of bits;
- identifying, based on the second determination, a first segment associated with a first fingerprint of the plurality of fingerprints;
- marking the first segment as unique;
- and
- deleting all of the segments associated with the plurality of fingerprints other than the first segment;
- and
- storing the deduplicated segments in the object storage.
Dependencies are inferred from the claim text.