Data storage system and method by shredding and deshredding

A method comprising:

• performing an allocation operation on initial data to allocate the initial data into allocated portions;
• selecting a first transformation operation for each allocated portion;
• applying the first transformation operation to the allocated portions;
• for each allocated portion, performing a redundancy operation on the transformed portions to generate a first number of shreds, the first number of shreds being greater than the number of transformed portions of the allocated portion;
• for each allocated portion, generating a tag for each shred, the tag comprising information identifying the allocation operation, the first transformation operation, and the redundancy operation applied previously to the initial data to generate the shred and indicating a shred order in which the allocation operation, the first transformation operation, and the redundancy operation were applied;
• reading the tag for at least a portion of the shreds to determine a deshred order that is the inverse of the shred order;
• for each allocated portion, in accordance with the deshred order, performing an operation inverse to the redundancy operation on a second number of shreds to output the transformed portions, the second number of shreds being smaller than the first number of shreds;
• for each allocated portion, in accordance with the deshred order, performing a transformation inverse to the first transformation operation on the transformed portions to generate the allocated portion;
• and
• in accordance with the deshred order, performing an operation inverse to the allocation operation on the allocated portions of the initial data to output the initial data.

A method for use with a block of data, the method comprising:

• selecting at least one transformation operation, an allocation operation, a redundancy operation, and a shred order in which to apply the at least one transformation operation, the allocation operation, and the redundancy operation to the block of data, the allocation operation being configured to divide data into a plurality of sub-blocks;
• applying the at least one transformation operation, the allocation operation, and the redundancy operation to the block of data in accordance with the shred order to obtain a plurality of shreds;
• generating a tag for each shred identifying the at least one transformation operation, the allocation operation, the redundancy operation applied to the block of data to obtain the shred, and the shred order;
• storing the plurality of shreds in at least one storage location;
• identifying a portion of plurality of shreds sufficient to recover the block of data, the portion being fewer than all of the shreds in the plurality of shreds;
• for each shred in the portion, reading the tag to determine an inverse operation for each of the at least one transformation operation, the allocation operation, and the redundancy operation applied to the block of data to obtain the shred, and to determine a deshred order inverse to the shred order for the shred;
• and
• for each shred in the portion, applying the inverse operation for the at least one transformation operation, the inverse allocation operation, and the inverse redundancy operation to the shred in accordance with the deshred order to obtain the block of data.

A computer-readable medium comprising instructions executable by a processor and when executed instructing the processor to:

• obtain a block of data;
• select at least one transformation operation, an allocation operation, a redundancy operation, and a shred order in which to the apply the at least one transformation operation, the allocation operation, and the redundancy operation to the block of data, the allocation operation being configured to divide data into a plurality of sub-blocks;
• apply the at least one transformation operation, the allocation operation, and the redundancy operation to the block of data in accordance with the shred order to obtain a plurality of shreds;
• generate a tag for each shred identifying the at least one transformation operation, the allocation operation, the redundancy operation applied to the block of data to obtain the shred, and the shred order;
• store the plurality of shreds in at least one storage location;
• identify a portion of plurality of shreds sufficient to recover the block of data, the portion being fewer than all of the shreds in the plurality of shreds;
• for each shred in the portion, read the tag to determine an inverse operation for each of the at least one transformation operation, the allocation operation, and the redundancy operation applied to the block of data to obtain the shred, and to determine a deshred order inverse to the shred order for the shred;
• and
• for each shred in the portion, apply the inverse operation for the at least one transformation operation, the inverse allocation operation, and the inverse redundancy operation to the shred in accordance with the deshred order to obtain the block of data.

A system for use with data, the system comprising:

• a storage location;
• a plurality of processors coupled to the storage location;
• a memory comprising instructions executable by the processors, the instructions when executed by one or more of the plurality of processors implementing a customizable shredder including an allocator, a first plurality of transformers, a tag generator, and a plurality of redundancy generators,
• the allocator being configured to perform an allocation operation allocating the data into a selected number of portions, the selected number being greater than one, and
• the first plurality of transformers being executable by the plurality of processors in parallel and during execution applying at least one transformation to the data,
• the plurality of redundancy generators being executable by the plurality of processors in parallel and during execution applying a redundancy operation to the data to add at least one redundant part of the data to the data, the memory further comprising instructions executable by the processors, the instructions when executed by one or more of the plurality of processors storing the data including the at least one redundant part in the storage location as a plurality of shreds, each shred storing a portion of the data,
• the tag generator being configured to generate a tag for each of the plurality of shreds, the tag comprising information identifying the allocation operation, the at least one transformation, and the redundancy operation applied to the portion of the data stored in the shred and a shred order in which the allocation operation, the at least one transformation, and the redundancy operation were applied to the portion of the data stored in the shred,
• the memory further comprising instructions executable by the processors, the instructions when executed by one or more of the plurality of processors implementing a deshredder including a tag reader, a plurality of inverse redundancy generators, a first plurality of inverse transformers, and a deallocator,
• for each of at least a portion the plurality of shreds, the tag reader configured to read the tag for the shred, identify the allocation operation, the at least one transformation, and the redundancy operation applied to the portion of the data stored in the shred and identify the shred order in which the allocation operation, the at least one transformation, and the redundancy operation were applied to the portion of the data stored in the shred, the memory further comprising instructions that when executed by one or more of the plurality of processors identifies an inverse allocation operation inverse to the allocation operation applied to the portion of the data stored in the shred, at least one inverse transformation inverse to the at least one transformation applied to the portion of the data stored in the shred, an inverse redundancy operation inverse to the redundancy operation applied to the portion of the data stored in the shred, and to execute the inverse allocation operation, the at least one inverse transformation, and the inverse redundancy operation in a deshred order inverse to the shred order,
• the plurality of inverse redundancy generators being executable by the plurality of processors in parallel and during execution applying the inverse redundancy operation to a portion of the plurality of shreds and outputting the portions of the data stored in each shred without the at least one redundant part, the portion of the plurality of shreds being fewer than all of the plurality of shreds, the outputted portions of the data stored in the portion of the plurality of shreds comprising the data,
• the first plurality of inverse transformers being executable by the plurality of processors in parallel and during execution applying the at least one inverse transformation to the portions of the data stored in the portion of the plurality of shreds, and
• the deallocator being configured to apply the inverse allocation operation to the portions of the data stored in the portion of the plurality of shreds.

A system for use with data, the system comprising:

• a storage location;
• a shredder in communication with the storage location and comprising a first plurality of logic circuits interconnected by one or more buses, a first portion of the first plurality of logic circuits being configured to implement an allocator, a second portion of the first plurality of logic circuits being configured to implement a plurality of transformers, a third portion of the first plurality of logic circuits being configured to implement a plurality of redundancy generators and a tag generator,
• the allocator performing an allocation operation allocating the data into a selected number of portions, the selected number being greater than one, and
• the plurality of transformers applying at least one transformation to the data,
• the plurality of redundancy generators applying a redundancy operation to the data to add redundancy to the data and storing the data in the storage location as a plurality of shreds, each shred storing a portion of the data,
• the tag generator being configured to generate a tag for each of the plurality of shreds, the tag comprising information identifying the allocation operation, the at least one transformation, and the redundancy operation applied to the portion of the data stored in the shred and a shred order in which the allocation operation, the at least one transformation, and the redundancy operation were applied to the portion of the data stored in the shred;
• and
• a deshredder in communication with the storage location and comprising a second plurality of logic circuits interconnected by one or more buses, a first portion of the second plurality of logic circuits being configured to implement a tag reader and a plurality of inverse redundancy generators, a second portion of the second plurality of logic circuits being configured to implement a first plurality of inverse transformers, and a third portion of the second plurality of logic circuits being configured to implement a deallocator,
• for each of at least a portion the plurality of shreds, the tag reader being configured to read the tag generated for the shred, identify the allocation operation, the at least one transformation, and the redundancy operation applied to the portion of the data stored in the shred and identify the shred order in which the allocation operation, the at least one transformation, and the redundancy operation were applied to the portion of the data stored in the shred, the tag reader being further configured to identify an inverse allocation operation inverse to the allocation operation applied to the portion of the data stored in the shred, at least one inverse transformation inverse to the at least one transformation applied to the portion of the data stored in the shred, and an inverse redundancy operation inverse to the redundancy operation applied to the portion of the data stored in the shred,
• the plurality of inverse redundancy generators applying the inverse redundancy operation to a portion of the plurality of shreds and outputting the portions of the data stored in each shred without the redundancy, the portion of the plurality of shreds being fewer than all of the plurality of shreds, the outputted portions of the data stored in the portion of the plurality of shreds comprising the data,
• the first plurality of inverse transformers applying the at least one inverse transformation to the portions of the data stored in the portion of the plurality of shreds;
• and
• the deallocator being configured to apply the inverse allocation operation to the portions of the data stored in the portion of the plurality of shreds.