Parity protection for data chunks in an object storage system

A method of generating a parity protection set in a distributed storage system containing a plurality of replicas of payload chunks, each of the replicas of payload chunks associated with a back-reference to one or more content manifests, the method comprising:

• identifying a content manifest to be protected;
• identifying a subset of all payload chunks referenced by the content manifest, wherein the subset is generated based on failure domain information;
• generating one or more parity chunks using the subset of payload chunks;
• generating a parity protection content manifest that refers to the parity protection set comprising the subset of payload chunks and the one or more parity chunks;
• generating a back-reference for the replica cited in the parity protection set for each payload chunk, wherein the back-reference is to the parity protection content manifest;
• erasing each back-reference to the content manifest for all remaining replicas of the subset of payload chunks, thereby enabling storage space in which the remaining replicas are stored to be used for future put requests;
• identifying a failure that invalidates a payload chunk in the parity protection set;
• and
• regenerating the invalidated payload chunk using the remaining payload chunks and the one or more parity protection chunks in the parity protection set.

A method of recovering a payload chunk in a distributed storage system, the method comprising:

• determining that a payload chunk cannot be retrieved or validated in response to a get transaction associated with a content manifest referring to the payload chunk;
• identifying a parity protection content manifest that refers to the payload chunk by using the content manifest or by analyzing back-references associated with the payload chunk;
• selecting a first parity protection set which includes the payload chunk from the parity protection contest manifest;
• attempting to regenerate the payload chunk using other payload chunks in the first parity protection set and one or more parity protection chunks in the parity protection set;
• and
• if the attempting step fails, repeating the attempting step with other parity protection sets which include the payload chunk until the payload chunk is regenerated.

A method of updating the retention tracking metadata for a parity protected chunk:

• replace all back-references to the parity protection content manifest with a verified back-reference to the corresponding manifest;
• generating one or more replicas of each payload chunk in the set of payload chunks;
• and
• for each of the one or more replicas of the payload chunk, generating one or more back-references to the same content manifests referred to by back-references for the payload chunk.

A method of creating a new version of a parity-protected object in a distributed storage system, comprising:

• generating a version manifest for the new version of the parity-protected object, wherein the version manifest refers to one or more payload chunks associated with the parity-protected object;
• generating a put request for each of the one or more payload chunks associated with the parity-protected object;
• in response to the put request for each of the one or more payload chunks associated with the parity-protected object, creating a speculative hold or extending a speculative hold on the payload chunk;
• and
• for each of the one or more payload chunks associated with the parity-protected object, adding a back-reference to the version manifest for the new version of the parity-protected object.

A method of creating a new version of a parity-protected object in a distributed storage system, comprising:

• generating a version manifest for the new version of the parity-protected object, wherein the version manifest refers to one or more payload chunks associated with the parity-protected object;
• generating a put request for each of the chunks representing modified portions of the object, but not initiating put transactions for unmodified retained chunks;
• creating a speculative hold on the prior version manifest to prevent the prior version manifest from being expunged until the process of protecting the retained chunks has been completed;
• and
• for each of the one or more payload chunks associated with the parity-protected object, adding a back-reference to the version manifest for the new version of the parity-protected object.