Doubly distributed erasure codes

A non-transitory computer-readable storage device storing computer executable instructions that when executed by a computer control the computer to perform a method for storing data in a hybrid data storage system, where the hybrid data storage system includes a plurality of storage devices having different initial read speeds, including a tape drive, a hard disk drive (HDD), a solid state device (SSD), or a shingled magnetic recording (SMR) device, the method comprising:

• accessing an erasure encoded object, where the erasure encoded object includes a plurality of code-words;
• chunking a code-word from the plurality of code-words into a set of code-word chunks, where the size of the set of code-word chunks is a function of the number storage devices in the hybrid data storage system on which the encoded object will be stored;
• generating a plurality of records by interleaving into a record a first code-word chunk from a first member of the plurality code-words with a second, different code-word chunk from a second, different member of the plurality of code-words, where a record includes metadata about the code-word chunks interleaved into the record;
• and
• storing the plurality of records on the plurality of storage devices, where storing the plurality of records includes storing a first record on a first storage device and storing a second, different record on a second, different storage device, where storing a record on a storage device includes storing code-word chunks from the same code-word at least a threshold physical distance apart on the storage device, where the threshold physical distance is based on a logical format of the storage device, an amount of data to be stored, a protection policy, a level of interleaving, a number of code-words, a symbol size, a data set size, a desired number of nines of durability, an environmental condition, or a number of storage devices in the plurality of storage devices.

The non-transitory computer-readable storage device of 19, the method comprising:

• upon determining which storage device in the plurality of storage devices has the fastest initial read speed;
• reading a record from the storage device with the fastest initial read speed;
• upon determining that a remaining storage device has attained a threshold read speed;
• reading a record from the remaining storage device;
• reconstructing the encoded object from the plurality of records;
• and
• upon detecting a full buffer or a full cache in a host system retrieving data from the plurality of storage devices, controlling the plurality of storage devices to continue delivering data to the host system for a threshold period of time.