Distributed erasure encoding in shared file system

A method for performing distributed erasure encoding in a storage system that includes an initiator device configured to read and write data in files stored in a plurality of storage nodes that are controlled by a file management system, the method comprising:

• generating a stripe by:
• identifying data to be stored in the plurality of storage nodes;
• dividing the data into a plurality of data segments;
• and
• performing an erasure encoding operation on the plurality of data segments to generate a parity segment,
• wherein the stripe includes the plurality of data segments and the parity segment;
• requesting, from the file management system, respective memory allocations in storage devices of the plurality of storage nodes for storing respective segments of the stripe;
• transmitting, to each of the plurality of storage nodes, a respective instruction to store a respective data segment or parity segment in the memory allocation on a respective storage device;
• performing a stripe checksum operation on bit values in the stripe and recording a stripe checksum result in each of the plurality of data segments and the parity segment;
• issuing a read request to the file management system for a second stripe of data;
• receiving a plurality of storage device memory locations that store segments of the second stripe from the file management system;
• requesting, from respective storage devices, respective segments of the second stripe stored at the received storage device memory locations;
• receiving a plurality of segments of the second stripe;
• determining a number of segments of the second stripe having matching stripe checksum results;
• determining whether the number of segments of the second stripe having matching stripe checksum results is a sufficient number of segments of the second stripe to recover data in the second stripe;
• and
• in response to determining that the number of segments of the second stripe having matching stripe checksum results is not a sufficient number of segments of the second stripe to recover data in the second stripe, determining that the second stripe of data has not been successfully read.

A distribution logic for performing distributed erasure encoding in a storage system that includes an initiator device configured to read and write data in files stored in a plurality of storage nodes that are controlled by a file management system, the distribution logic comprising:

• management logic configured to generate a stripe by:
• identifying data to be stored in the plurality of storage nodes;
• and
• dividing the data into a plurality of data segments;
• erasure coding logic configured to perform an erasure encoding operation on the plurality of data segments to generate a parity segment,
• wherein the stripe includes the plurality of data segments and the parity segment;
• mapping logic configured to request, from the file management system, respective memory allocations in storage devices of the plurality of storage nodes for storing respective segments of the stripe;
• and
• I/O logic configured to (i) transmit, to each of the plurality of storage nodes, a respective instruction to store a respective data segment or parity segment in the memory allocation on a respective storage device;
• (ii) receive respective acknowledgements from each of a subset of the plurality of storage nodes indicating that a respective segment of the stripe was successfully stored in;
• (iii) determine whether the acknowledged number of segments of the stripe that were successfully stored in a respective storage node is a sufficient number of segments of the stripe to recover the data;
• and (iv) in response to determining that the acknowledged number of segments of the stripe that were successfully stored in a respective storage node is not a sufficient number of segments of the stripe to recover the data, determine that the data was not successfully stored in the plurality of storage nodes.

An distribution logic for performing distributed erasure encoding in a storage system that includes an initiator device configured to read and write data in files stored in a plurality of storage nodes that are controlled by a file management system, the distribution logic comprising:

• erasure coding logic being configured to:
• identify a plurality of data segments of data stored in the plurality of storage nodes;
• perform an erasure encoding operation on the plurality of data segments to generate a parity segment, wherein a stripe includes the plurality of data segments and the parity segment;
• and
• perform a stripe checksum operation on bit values in the stripe and recording a stripe checksum result in each of the plurality of data segments and the parity segment;
• mapping logic that is configured to:
• issue a read request to the file management system for a second stripe of data;
• and
• receive a plurality of storage device memory locations that store segments of the second stripe from the file management system;
• and
• I/O logic that is configured to:
• request, from respective storage devices, respective segments of the second stripe stored at the received storage device memory locations;
• receive a plurality of segments of the second stripe;
• determine a number of segments of the second stripe having matching stripe checksum results;
• determine whether the number of segments of the second stripe having matching stripe checksum results is a sufficient number of segments of the second stripe to recover data in the second stripe;
• and
• in response to determining that the number of segments of the second stripe having matching stripe checksum results is not a sufficient number of segments of the second stripe to recover data in the second stripe, determine that the second stripe of data has not been successfully read.