Hierarchical wide spreading of distributed storage

A non-transitory machine readable medium having stored thereon instructions for performing a method, wherein the instructions comprise machine executable code which when executed by at least one machine, causes the machine to:

• in response to a request to write an object, divide the object into multiple source data segments;
• generate redundancy data for the multiple source data segments using a rateless erasure encoding;
• associate a respective subset of the redundancy data with each of the multiple source data segments, wherein each subset of redundancy data and associated source data segment form an encoded segment;
• define multiple segment-level fragments within each of the encoded segments;
• transmit each of the encoded segments to a selected one of multiple storage nodes, wherein each of the selected storage nodes are selected based on a determined storage layout of the encoded segments across the multiple storage nodes;
• and
• for each of the encoded segments received at each of the selected storage nodes,
• generate one or more protection fragments based on redundancy data generated from the segment-level fragments;
• and
• store the segment-level fragments and corresponding protection fragments across multiple storage media devices managed by the selected storage node.

A computing device comprising:

• a memory comprising machine readable media that contains machine executable code;
• a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to:
• in response to a request to write an object, divide the object into multiple source data segments;
• generate redundancy data for the multiple source data segments using a rateless erasure encoding;
• associate a respective subset of the redundancy data with each of the multiple source data segments, wherein each subset of redundancy data and associated source data segment form an encoded segment;
• define multiple segment-level fragments within each of the encoded segments;
• transmit each of the encoded segments to a selected one of multiple storage nodes, wherein each of the selected storage nodes are selected based on a determined storage layout of the encoded segments across the multiple storage nodes;
• and
• for each of the encoded segments received at each of the selected storage nodes,
• generate one or more protection fragments based on redundancy data generated from the segment-level fragments;
• and
• store the segment-level fragments and corresponding protection fragments across multiple storage media devices managed by the selected storage node.