Distributed Storage System Data Management And Security

A method of data retrieval from remote storage nodes, the method comprising:

• accessing, by one or more processors configured to execute code stored in non-transitory processor readable media, file metadata references within a local cache or within remote storage nodes;
• receiving, by the one or more processors, a plurality of packages from remote storage nodes by metadata references, each of the packages contain file metadata;
• receiving, by the one or more processors, a plurality of other packages containing encoded file segments from storage nodes by data references, wherein the encoded file segments are obtained at least partly from file metadata;
• reconstructing, by the one or more processors, file data from the packages as a function of metadata representing parameters associated with an encoding scheme and file splitting scheme.

A system for distributing data of a plurality of files over a plurality of respective remote storage nodes, the method comprising:

• one or more processors in electronic communication with non-transitory processor readable storage media,
• one or more software modules comprising executable instructions stored in the storage media, wherein the one or more software modules are executable by the one or more processors and include:
• a fragmentation module that configures the one or more processors to split into segments the data of the plurality of files;
• an encoding module that configures the one or more processors to encode each segment into a number of codeword chunks, wherein none of the codeword chunks contains any of the segments and to package each codework chunk with at least one encoding parameter and identifier;
• a configuration module that configures the one or more processors to generate metadata for at least one file of the plurality of files and metadata for related segments of the at least one file, wherein the metadata for the at least one file contains information to reconstruct the at least one file from the segments, and metadata for the related segments contains information for reconstructing the related segments from corresponding packages;
• wherein the metadata is encoded into at least one package, wherein the encoding corresponds to a respective security level and a protection against storage node failure;
• a load balancing module that configures the one or more processors to assign a plurality of packages to remote storage nodes, wherein the step of assigning corresponds to optimized workload distribution;
• including as a function of available network bandwidth;
• a control module that configures the one or more processors to transmit each of the packages to at least one respective storage node, and to retrieve at least one of the plurality of files, as a function iteratively accessing and retrieving the packages of metadata and file data.