Chunk retention in a distributed object storage system using stream sessions and stream session backreferences

A method for storing chunks of data objects in a distributed object storage system, the method comprising:

• encoding forward-references from version manifests to the chunks in a manifest tracking subsystem which stores the version manifests;
• and
• encoding stream sessions backreferences into metadata associated with the chunks, wherein the stream session backreferences each identifies a stream session having a finite retention epoch, and wherein each stream session identifies a set of one or more objects being edited and a source providing edits during the retention epoch;
• requesting by the source a put of a version manifest to the system during the retention epoch;
• reading chunk references that are contained, directly or indirectly, in the version manifest;
• using the chunk references to determine inherited chunks, wherein the inherited chunks are chunks from a reference version manifest that is identified by the stream session;
• and
• using the chunk references to determine put-during-session chunks,
• wherein the put-during-session chunks are chunks that were put previously during the retention epoch.

A system for storing chunks of data objects, the system comprising:

• one or more servers encoding forward-references from version manifests to the chunks in a manifest tracking subsystem which stores the version manifests;
• and
• one or more servers encoding stream sessions backreferences into metadata associated with the chunks, wherein the stream session backreferences each identifies a stream session having a finite retention epoch, and wherein each stream session identifies a set of one or more objects being edited and a source providing edits during the retention epoch,
• wherein the source requests a put of a version manifest to the system during the retention epoch, chunk references that are contained, directly or indirectly, in the version manifest are read, and the chunk references are used to determine inherited chunks, the inherited chunks being chunks from a reference version manifest that is identified by the stream session, and
• wherein a touch operation is performed to add stream session backreferences to the inherited chunks.

A distributed storage system for storing chunks of data objects, the system comprising:

• one or more servers encoding forward-references from version manifests to the chunks in a manifest tracking subsystem which stores the version manifests;
• and
• one or more servers encoding stream sessions backreferences into metadata associated with the chunks, wherein the stream session backreferences each identifies a stream session having a finite retention epoch, and wherein each stream session identifies a set of one or more objects being edited and a source providing edits during the retention epoch,
• wherein the source requests a put of a version manifest to the system during the retention epoch, chunk references that are contained, directly or indirectly, in the version manifest are read, and the chunk references are used to determine inherited chunks, the inherited chunks being chunks from a reference version manifest that is identified by the stream session, and
• wherein the chunk references are used to determine put-during-session chunks, and the put-during-session chunks are chunks that were put previously during the retention epoch.