Data redirector for scale out

An apparatus for distributing data from a source to a plurality of deduplication blockpools, the apparatus comprising:

• a processor;
• a memory configured to store a binary large object (BLOB) having a first size, where the BLOB includes a plurality of blocklets, a blocklet having a hash value and a second size, where the second size is smaller than the first size;
• an input/output (I/O) interface;
• a set of circuits;
• and
• an interface configured to connect the processor, the memory, the I/O interface, and the set of circuits, the set of circuits including an affinity circuit, a rule set selection circuit, a first rule set circuit, a second rule set circuit, and a data redirection circuit;
• the affinity circuit configured to:
• determine which member of the plurality of deduplication blockpools was the most recently selected target blockpool from among the plurality of deduplication blockpools;
• assign the most recently selected target blockpool as a current affinity blockpool;
• the rule set selection circuit configured to:
• determine the number of blocklets in the plurality of blocklets;
• and
• select the first rule set circuit or the second rule set circuit to select a target blockpool, where the rule set selection circuit selects the first rule set circuit or the second rule set circuit based on the number of blocklets in the plurality of blocklets;
• the first rule set circuit configured to select a target blockpool from among the plurality of deduplication blockpools according to an affinity rule or a randomness rule;
• the second rule set circuit configured to select a target blockpool from among the plurality of deduplication blockpools according to a balancing rule, an affinity rule, a size rule, or a randomness rule;
• and
• the data redirection circuit configured to provide the BLOB to the target blockpool.

A non-transitory computer-readable storage device storing computer-executable instructions that when executed by a computer control the computer to perform a method for distributing data from a source to a plurality of deduplication blockpools, the method comprising:

• accessing a binary large object (BLOB) having a first size, the BLOB including a plurality of blocklets, a blocklet having a second, different size that is smaller than the first size, a blocklet further having a hash;
• determining whether the plurality of deduplication blockpools supports BLOB-level redirection, or blocklet-level redirection;
• upon determining that the plurality of deduplication blockpools supports blocklet-level redirection:
• selecting a target blockpool for a member of the plurality of blocklets based on a value of the hash of the member the plurality of blocklets modulo P, where P is the number of deduplication blockpools in the plurality of deduplication blockpools;
• and
• providing the blocklet to the target blockpool;
• upon determining that the plurality of deduplication blockpools supports BLOB-level redirection:
• determining the number of blocklets in the BLOB;
• upon determining that the number of blocklets in the BLOB is less than a threshold number of blocklets:
• selecting a target blockpool from among the plurality of deduplication blockpools according to a first rule set;
• upon determining that the number of blocklets in the BLOB is at least the threshold number of blocklets:
• selecting a target blockpool from among the plurality of deduplication blockpools according to a second, different rule set, where the second, different rule set computes a fitness value for a member of the plurality of deduplication blockpools, and that selects a target blockpool from among the plurality of deduplication blockpools based, at least in part, on the fitness value, where the fitness value is a function of a correspondence between a subset of the plurality of blocklets and blocklets stored in a member of the plurality of deduplication blockpools, where the subset of the plurality of blocklets contains fewer members than the plurality of blocklets, and further based on a ratio of the remaining storage capacity of the member of the plurality of deduplication blockpools to the remaining storage capacity of the member of the plurality of deduplication blockpools having the largest remaining storage capacity;
• and
• providing the BLOB to the target blockpool.