Supporting deduplication in object storage using subset hashes

A non-transitory machine-readable medium having instructions stored thereon which, when executed by a processor, cause the processor to:

• receive a plurality of hashes of a plurality of blocks of a plurality of log segments from a software defined data center, wherein each block corresponds to a respective logical address;
• associate each of the plurality of logical addresses with a respective sequentially-allocated chunk identifier in a logical map that stores logical associations between chunk identifiers and logical addresses in an order in which the log segments are received, wherein contiguous chunks that point to a same log segment and have a same reference count are merged into a single tuple;
• determine a subset hash, the subset hash comprising a hash of a subset of the plurality of blocks, wherein the subset corresponds to a contiguous range of the plurality of logical addresses;
• perform a search of a hash map for the subset hash to determine if the subset hash is a duplicate, wherein the hash map stores associations between chunk identifiers and subset hashes;
• and
• deduplicate the subset of the plurality of blocks responsive to a determination that the subset hash is a duplicate.

A system, comprising:

• a processor;
• and
• a memory having instructions stored thereon which, when executed by the processor, cause the processor to:
• receive a plurality of hashes of a plurality of blocks of a plurality of log segments from a software defined data center, wherein each block corresponds to a respective logical address;
• associate each of the plurality of logical addresses with a respective sequentially-allocated chunk identifier in a logical map that stores logical associations between chunk identifiers and logical addresses in an order in which the log segments are received, wherein contiguous chunks that point to a same log segment and have a same reference count are merged into a single tuple;
• determine a subset hash, the subset hash comprising a hash of a subset of the plurality of blocks, wherein the subset corresponds to a contiguous range of the plurality of logical addresses;
• perform a search of a hash map for the subset hash to determine if the subset hash is a duplicate, wherein the hash map stores associations between chunk identifiers and subset hashes;
• and
• deduplicate the subset of the plurality of blocks responsive to a determination that the subset hash is a duplicate.