Erasure coding and replication in storage clusters

A method of reading a digital object from a storage cluster having a plurality of computer nodes, said method comprising:

• receiving, at one of said computer nodes within said storage cluster, a request from a client application that includes a unique identifier for said digital object;
• determining whether said digital object is stored within said storage cluster using replication or erasure coding;
• when it is determined that said digital object is stored using erasure coding, reading a manifest stored on a computer nodes of said storage cluster, said manifest being identified by said unique identifier;
• identifying a plurality of segments within said storage cluster using unique segment identifiers found within said manifest;
• reconstructing said digital object using said segments and an erasure coding algorithm;
• and
• returning said digital object to said client application.

A method of recovering from a failed disk, said method comprising:

• detecting within a storage cluster having a plurality of computer nodes that a first disk of one of said plurality of computer nodes has failed;
• scanning a persistent storage area of a second disk of said storage cluster to find a unique identifier of said failed disk, said unique identifier in association with a digital stream of said storage cluster;
• determining whether said digital stream is stored within said storage cluster using replication or erasure coding;
• when it is determined that said digital stream is stored using erasure coding, identifying a missing segment previously stored on said failed disk;
• locating a plurality of other segments within said storage cluster, said plurality of other segments including said digital stream;
• regenerating said missing segment previously stored on said failed disk using said plurality of other segments and an erasure coding algorithm;
• and
• storing said regenerated segment on a computer node of said storage cluster.

A method of relocating a segment within a storage cluster, said method further comprising:

• identifying a segment on a first disk of a computer node within a storage cluster, said segment being one of a plurality of segments representing a digital object stored within said storage cluster;
• relocating said segment from said first disk to a second disk of said storage cluster, said second disk being identified by a second unique disk identifier;
• retrieving from metadata of said segment a unique identifier for a sibling segment of said plurality of segments, said sibling segment including in its metadata a first unique disk identifier for said first disk;
• locating said sibling segment within said storage cluster using said unique identifier;
• and
• replacing said first unique disk identifier for said first disk within said metadata of said sibling segment with a second unique disk identifier for said second disk, whereby said metadata of said sibling segment indicates said second disk to where said segment has been relocated.

A method of converting a digital object within a storage cluster, said method comprising:

• storing said digital object within said storage cluster on a single disk of a computer node as a continuous stream of bits, said digital object having a unique identifier within said storage cluster;
• after said storing, identifying metadata of said storage cluster indicating a requirement to convert said digital object to an erasure coding storage format;
• reading said digital object from said single disk using said computer node of said storage cluster;
• writing said digital object to a plurality of disks of said storage cluster using said erasure coding storage format;
• and
• retaining said unique identifier for said digital object written in said erasure coding storage format, whereby a client application may retrieve said digital object written in said erasure coding storage format using said unique identifier.

A method of converting a digital object within a storage cluster, said method comprising:

• storing said digital object within said storage cluster in a first erasure coding storage format, said digital object having a unique identifier within said storage cluster;
• after said storing, identifying metadata of said storage cluster indicating a requirement to convert said digital object to a second erasure coding storage format;
• reading said digital object from said storage cluster using a computer node of said storage cluster;
• writing said digital object to said storage cluster using said second erasure coding storage format;
• and
• retaining said unique identifier for said digital object written in said second erasure coding storage format, whereby a client application may retrieve said digital object written in said second erasure coding storage format using said unique identifier.

A method of converting a digital object within a storage cluster, said method comprising:

• storing said digital object within said storage cluster in an erasure coding storage format, said digital object having a unique identifier within said storage cluster;
• after said storing, identifying metadata of said storage cluster indicating a requirement to convert said digital object to a storage format using replication;
• reading said digital object from said storage cluster using a computer node of said storage cluster;
• writing said digital object to a single disk of a computer node of said storage cluster as a continuous stream of bits;
• and
• retaining said unique identifier for said digital object written as said continuous stream of bits, whereby a client application may retrieve said digital object written as said continuous stream of bits using said unique identifier.

A method of copying a digital object from a source storage cluster to a target storage cluster, said method comprising:

• receiving an instruction at a target node of said target storage cluster to copy said digital object from said source storage cluster to said target storage cluster, said instruction including an address of a source node of said source storage cluster;
• determining a target conversion format into which said digital object will be stored in said target storage cluster;
• reading said digital object from said source storage cluster, said digital object being stored in a source conversion format and having a unique identifier within said source storage cluster;
• and
• storing said digital object in said target storage cluster using said target conversion format, said digital object being stored using said unique identifier.