Scheduling requests from data sources for efficient data decoding

A method, comprising:

• performing, by one or more computers:
• receiving a request from a client for a data object stored at a storage system according to an erasure encoding technique, wherein a respective block of the data object can be reconstructed from at least a particular number of a plurality of corresponding blocks stored at a plurality of different storage devices of the storage system;
• requesting, over a network, a set of corresponding blocks for a plurality of blocks of the data object from at least the particular number of the storage devices;
• storing the corresponding blocks of the data object received from the storage devices in an input buffer as the corresponding blocks are received in response to the requesting;
• in response to receiving at least the particular number of corresponding blocks from the storage devices for a particular block of the data object:
• removing the at least the particular number of corresponding blocks from the input buffer;
• and
• decoding the particular number of corresponding blocks into a reconstructed block for the particular block of the data object;
• and
• in response to an amount of free space in the input buffer reaching a threshold amount:
• determining, based at least in part on an amount of free space in the input buffer, a number of blocks to request from the at least the particular number of the storage devices, and
• requesting over the network from at least the particular number of the storage devices that store blocks of the respective data object, a subsequent set of corresponding blocks for another plurality of blocks of the data object, wherein the threshold amount is less than a buffer size of the input buffer such that at least some blocks are stored in the input buffer when the threshold amount is reached.

A non-transitory, computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to:

• implement a storage manager configured to:
• receive a request from a client for a data object stored at a storage system according to an erasure encoding technique, wherein a respective block of the data object can be reconstructed from at least a particular number of a plurality of corresponding blocks stored at a plurality of different storage devices of the storage system;
• request, over a network, a set of corresponding blocks for a plurality of blocks of the data object from at least the particular number of the storage devices;
• store the corresponding blocks of the data object received from the storage devices in an input buffer as the corresponding blocks are received in response to the requesting;
• in response to receiving at least the particular number of corresponding blocks from the storage devices for a particular block of the data object:
• remove the at least the particular number of corresponding blocks from the input buffer;
• and
• decode the particular number of corresponding blocks into a reconstructed block for the particular block of the data object;
• and
• in response to an amount of free space in the input buffer reaching a threshold amount:
• determine, based at least in part on an amount of free space in the input buffer, a number of blocks to request from the at least the particular number of the storage devices, and
• request over the network from at least the particular number of the storage devices that store blocks of the respective data object, a subsequent set of corresponding blocks for another plurality of blocks of the data object, wherein the threshold amount is less than a buffer size of the input buffer such that at least some blocks are stored in the input buffer when the threshold amount is reached.