Content-based post-process data deduplication

A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the program code embodying a method for implementing content-based post-process data deduplication, the method comprising:

• receiving a write request comprising write data to be persisted to a storage system;
• sampling a portion of the write data;
• executing one or more analyses on the sampled portion in order to determine whether the write data is a good deduplication candidate that is likely to contain redundancies which can be eliminated via data deduplication, the one or more analyses including a signature-based analysis that involves comparing the sampled portion with a plurality of data signatures stored in a signature database, each data signature in the plurality of data signatures comprising a sequence of bytes at certain data block offsets that is found in one or more types of storage objects that are good deduplication candidates;
• in response to determining that the one or more analyses indicate the write data is a good deduplication candidate, causing the write data to be persisted to a staging storage component of the storage system;
• in response to determining that the one or more analyses indicate the write data is not a good deduplication candidate, causing the write data to be persisted to a primary storage component of the storage system;
• sampling data blocks from the primary storage component and the secondary storage component that are fetched in response to read requests initiated by storage clients;
• determining, based on the sampled data blocks, an accuracy of the computer system in identifying good deduplication candidates;
• and
• if the determined accuracy is below a predefined threshold:
• generating one or more new or refined data signatures via an analysis of the sampled data blocks;
• and
• saving the new or refined data signatures to the signature database.

A computer system comprising:

• a processor;
• and
• a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
• receive a write request comprising write data to be persisted to a storage system;
• sample a portion of the write data;
• execute one or more analyses on the sampled portion in order to determine whether the write data is a good deduplication candidate that is likely to contain redundancies which can be eliminated via data deduplication, the one or more analyses including a signature-based analysis that involves comparing the sampled portion with a plurality of data signatures stored in a signature database, each data signature in the plurality of data signatures comprising a sequence of bytes at certain data block offsets that is found in one or more types of storage objects that are good deduplication candidates;
• in response to determining that the one or more analyses indicate the write data is a good deduplication candidate, cause the write data to be persisted to a staging storage component of the storage system;
• in response to determining that the one or more analyses indicate the write data is not a good deduplication candidate, cause the write data to be persisted to a primary storage component of the storage system;
• sample data blocks from the primary storage component and the secondary storage component that are fetched in response to read requests initiated by storage clients;
• determine, based on the sampled data blocks, an accuracy of the computer system in identifying good deduplication candidates;
• and
• if the determined accuracy is below a predefined threshold:
• generate one or more new or refined data signatures via an analysis of the sampled data blocks;
• and
• save the new or refined data signatures to the signature database.