Systems, methods, and computer program products providing change logging in a deduplication process

A computer program product having a non-transitory computer readable medium tangibly recording computer program logic for administrating data backup in a network storage system that has a primary storage subsystem and backup storage subsystem, the computer program product comprising:

• code to transfer a plurality of data blocks to the backup storage subsystem;
• code to generate a data log for the plurality of data blocks, the data log having a first plurality of entries, each entry including block name and block fingerprint for a respective block;
• code to transfer information relating the plurality of data blocks to the plurality of files from the primary storage subsystem to the secondary storage subsystem;
• code to generate a reference log using the information relating the plurality of data blocks to the plurality of files, the reference log including a second plurality of entries, each entry including block name for one of the data blocks, metadata describing one of the files, and a number representing an order of the one of the data blocks within the one of the files;
• code to merge the data log and the reference log to create a change log with at least one entry for each of the data blocks, wherein the at least one entry includes data of a first type from the data log and corresponding to the data block and data of a second type from the reference log and corresponding to the data block, and wherein the first type and the second type are different;
• and
• code to perform deduplication by using the change log to check for duplication within the plurality of data blocks.

A system comprising:

• a storage subsystem;
• and
• one or more processors operatively coupled with the storage subsystem to administer data backup and deduplication, the one or more processors executing code to perform the following operations:
• processing a plurality of files for backup up at the storage subsystem in a first phase and a second phase, in which the first phase includes transferring a plurality of data blocks and creating fingerprints for the data blocks, and in which the second phase includes transferring metadata relating the plurality of blocks to the plurality of files;
• creating a first log based on data received during the first phase, wherein the first log includes entries for each of the data blocks, each entry of the first log listing a block name and a fingerprint;
• creating a second log based on data received during the second phase, wherein the second log includes entries associating the plurality of data blocks with the plurality of files, each entry listing a block name, metadata describing a file, and a number identifier of a block within a file;
• and
• merging the first log and second log to generate a change log including entries associating a respective file with a respective number identifier and a fingerprint, wherein at least one of the entries of the change log includes data for a block of the plurality of data block from the first log and data for the block from the second log, and wherein the data from the first log and the data from the second log are different.