Methods and systems for improved throughput performance in a distributed data de-duplication environment

A method of data storage in a data de-duplication system comprising:

• controlling a local client node to parse a stream of data received at the local client node into a set of variable length blocks at the local client node;
• determining, at the local client node, a code that represents a block of data parsed from the stream, the code being a hash of the block;
• controlling the local chant node to send the code representing the block of data to a server, where the code is sent over a network;
• receiving, at the local client node, from the server, a notification that the block is unique as identified by the server in response to examining the code;
• in response to receiving the notification from the server at the local client node, controlling the local client node to write the block identified as a unique block by the notification to storage associated with the local client node;
• in response to receiving the notification from the server at the local client node, controlling the local client node to write the code associated with the unique block to a file at the local client node, the file being located on a storage device at the local client node, the file being configured to facilitate performing uniqueness comparisons at the local client node;
• updating metadata at the server, where the metadata is associated with the existence of the unique block, the code associated with the unique block, and the location of the unique block, and
• updating an index at the server with information concerning the existence of the unique block, the code associated with the unique block, and the location of the unique block.