Hierarchical chunking of objects in a distributed storage system

A method for managing placement of object replicas in a distributed storage system, comprising:

• at a first instance of the distributed storage system, having one or more processors and memory, wherein the memory stores one or more programs for execution by the one or more processors:
• opening one or more journals for storage of object chunks, wherein each respective journal is associated with a single respective placement policy;
• receiving a first object comprising at least a first object chunk, wherein the first object is associated with a first placement policy, and wherein the first object chunk comprises a first plurality of blocks;
• storing the first plurality of blocks in a first journal whose associated placement policy matches the first placement policy, wherein the first journal stores only blocks for objects whose placement policies match the first placement policy;
• storing global metadata for the first object, wherein the global metadata includes a first list of object chunks corresponding to the first object, and wherein the first list includes an identifier of the first object chunk;
• storing local metadata for the first object chunk, wherein the local metadata includes a block list identifying each block of the first plurality of blocks, and wherein the local metadata is associated with the first journal;
• for the first journal, repeating the receiving and storing operations for a first plurality of objects whose associated placement policies match the first placement policy, until a first termination condition occurs;
• after the first termination condition occurs, closing the first journal, thereby preventing any additional blocks from being stored in the first journal;
• and
• replicating the first journal to a second instance of the distributed storage system in accordance with the first placement policy, wherein the global metadata is updated to reflect the replication, whereas the local metadata is unchanged by the replication.

A computer system for managing placement of object replicas in a distributed storage system having a plurality of instances, each respective instance comprising:

• one or more processors;
• memory;
• and
• one or more programs stored in the memory, the one or more programs comprising instructions executable by the one or more processors for:
• opening one or more journals for storage of object chunks, wherein each respective journal is associated with a single respective placement policy;
• receiving a first object comprising at least a first object chunk, wherein the first object is associated with a first placement policy, and wherein the first object chunk comprises a first plurality of blocks;
• storing the first plurality of blocks in a first journal whose associated placement policy matches the first placement policy, wherein the first journal stores only blocks for objects whose placement policies match the first placement policy;
• storing global metadata for the first object, wherein the global metadata includes a first list of object chunks corresponding to the first object, and wherein the first list includes an identifier of the first object chunk;
• storing local metadata for the first object chunk, wherein the local metadata includes a block list identifying each block of the first plurality of blocks, and wherein the local metadata is associated with the first journal;
• for the first journal, repeating the receiving and storing operations for a first plurality of objects whose associated placement policies match the first placement policy, until a first termination condition occurs;
• after the first termination condition occurs, closing the first journal, thereby preventing any additional blocks from being stored in the first journal;
• and
• replicating the first journal to a second instance of the distributed storage system in accordance with the first placement policy, wherein the global metadata is updated to reflect the replication, whereas the local metadata is unchanged by the replication.