Hardware acceleration of commonality factoring with removable media

A method for commonality factoring for storing data with a removable storage cartridge, comprising:

• receiving, at an expansion module removably coupled to a host computer, an original data stream, wherein the expansion module comprises:
• a chunking module, and
• a hashing module,
• wherein the hashing module and the chunking module are configured in a pipeline architecture such that at least a portion of input to the hashing module comprises output from the chunking module;
• at the chunking module, breaking the original data stream into a plurality of chunks;
• and
• forwarding the chunks toward the hashing module, wherein the hashing module performs the following steps:
• calculating an identifier for each forwarded chunk;
• storing the identifiers;
• and
• determining, based on the identifiers, whether each chunk is unique;
• and
• forwarding at least one of the unique chunks and the identifier to the removable storage cartridge, wherein the removable storage cartridge includes a storage drive;
• and
• further wherein multiple data paths in parallel are utilized and the multiple data paths comprise a single data stream split across multiple instances by truncating the single data stream.

An expansion card for commonality factoring for storing data with a removable storage cartridge, comprising:

• a chunking module configured to:
• receive an original data stream from the host computer, and
• break the original data stream into a plurality of chunks, wherein the expansion card is configured to:
• be removably coupled to a host computer and the removable storage cartridge, and
• store data on the removable storage cartridge;
• and
• a hashing module coupled to the chunking module in a pipeline fashion such that at least a portion of input to the hashing module comprises output from the chunking module, wherein the hashing module is configured to:
• receive the plurality of chunks from the chunking module,
• calculate an identifier for each of the received chunks,
• determine, based on the identifiers, if each chunk is unique, and
• store the unique chunks on the removable storage cartridge, wherein multiple data paths in parallel are utilized and the multiple data paths comprise a single data stream split across multiple instances by truncating the single data stream.