Methods and systems for providing read-optimized scalable offline de-duplication for blocks of data

A system for read-optimized de-duplicating of blocks of data, comprising:

• at least one processor of a server, the processor configured to:
• for each page of a newly received block of data, compute a page hash and assign a page context ID, wherein the page context ID is computed as a minimum of hashes of all pages in a buffer containing the page and a fixed amount of pages preceding and following the page in the block of data;
• for each page, compute an index of a page node for storing the page based on the page context ID assigned to the page and a number of page nodes in a cluster and select the page node that matches the computed index of the page node;
• for each page, determine whether the selected page node contains a copy of the page;
• for each page, when the selected page node contains the copy of the page, add a page reference to a block descriptor with the page being referenced by the page hash augmented by the page context ID;
• for each page, when the selected page node does not contain the copy of the page, store the page in plain storage;
• and
• for the block of data, store the block descriptor in a block node of a de-duplication engine, the block of data being referenced by a block hash and the block descriptor containing at least a list of pages of the block stored in page nodes of the de-duplication engine.

A non-transitory computer readable medium storing thereon computer executable instructions for read-optimized de-duplication of blocks of data, including instructions for:

• for each page of a newly received block of data, computing a page hash and assigning a page context ID, wherein the page context ID is computed as a minimum of hashes of all pages in a buffer containing the page and a fixed amount of pages preceding and following the page in the block of data;
• for each page, computing an index of a page node for storing the page based on the page context ID assigned to the page and a number of page nodes in a cluster and selecting the page node that matches the computed index of the page node;
• for each page, determining whether the selected page node contains a copy of the page;
• for each page, when the selected page node contains the copy of the page, adding a page reference to a block descriptor with the page being referenced by the page hash augmented by the page context ID;
• for each page, when the selected page node does not contain the copy of the page, storing the page in plain storage;
• and
• for the block of data, storing the block descriptor in a block node of a de-duplication engine, the block of data being referenced by a block hash and the block descriptor containing at least a list of pages of the block stored in page nodes of the de-duplication engine.