System and method for optimized signature comparisons and data replication

A method for generating a remote backup copy on a remote network node of a dataset located on a source network node, the method comprising:

• receiving a command to duplicate the dataset located on the source network node to the remote network node, the dataset comprising signatures of data and metadata in a tree structured file system, wherein the data and metadata are stored as objects in an object store, each object having a content-derived signature;
• in response to receipt of the command, accessing, on the source network node, signatures of the data and the metadata of the dataset in a previously stored snapshot of the dataset and the signatures of the data and the metadata of the dataset in a current snapshot of the dataset, wherein the signatures of the data and the metadata of the dataset in the previously stored snapshot are in a tree structured file system, and wherein the previously stored snapshot corresponds to a remote snapshot of the dataset previously stored at the remote network node;
• performing, at the source network node, a local signature comparison between the previously stored snapshot and the current snapshot to identify one or more signatures that do not match between the previously stored snapshot and the current snapshot, wherein the local signature comparison is an iterative top-down comparison beginning at the root of a tree of the tree structured file system, and wherein the signatures that do not match have at least one of different content or different placement in the tree;
• sending the signatures that do not match to the remote network node;
• receiving, from the remote network node, a response of whether the signatures that do not match exist at the remote network node;
• and
• based on a determination that the signatures that do not match do not exist at the remote network node, sending data associated with the signatures that do not match to the remote network node for incorporation within a current remote snapshot.

A non-transitory computer-readable storage medium having stored therein instructions which, when executed by a processor, cause the processor to:

• receive a command to duplicate a dataset located on a source network node to a remote network node, the dataset comprising signatures of data and metadata in a tree structured file system, wherein the data and metadata are stored as objects in an object store, each object having a content-derived signature;
• in response to receipt of the command, access, on the source network node, signatures of the data and the metadata of the dataset in a previously stored snapshot of the dataset and the signatures of the data and the metadata of the dataset in a current snapshot of the dataset, wherein the signatures of the data and the metadata of the dataset in the previously stored snapshot are in a tree structured file system, and wherein the previously stored snapshot corresponds to a remote snapshot of the dataset previously stored at the remote network node;
• perform, at the source network node, a local signature comparison between the previously stored snapshot and the current snapshot to identify one or more signatures that do not match between the previously stored snapshot and the current snapshot, wherein the local signature comparison is an iterative top-down comparison beginning at the root of a tree of the tree structured file system, and wherein the signatures that do not match have at least one of different content or different placement in the tree;
• send the signatures that do not match to the remote network node;
• receive, from the remote network node, a response of whether the signatures that do not match exist at the remote network node;
• and
• based on a determination that the signatures that do not match do not exist at the remote network node, send data associated with the signatures that do not match to the remote network node for incorporation within a current remote snapshot.