Systems and methods for sharding based on distributed inverted indexes

A server of a cloud-based storage system, the server comprising:

• a processor;
• and
• a memory coupled with and readable by the processor and storing therein a set of instructions which, when executed by the processor, causes the processor to distribute data across a plurality of storage shards by:
• generating a file key for each file of a plurality of files stored in a plurality of physical shards, each physical shard maintained by a node of a plurality of nodes in one or more clusters, wherein the file key comprises a hash of an enterprise identifier (enterprise ID) for an entity to which the creator of the file is a member, a hash of a folder identifier (folder ID) for a location in which the file is stored, and a hash of a file identifier (file ID) uniquely identifying the file;
• sorting the generated file keys for each file of the plurality of files into an ordered list;
• logically partitioning the ordered list into a plurality of logical shards;
• mapping each logical shard of the plurality of logical shards to one of the plurality of physical shards;
• identifying a last key value for each logical shard in the partitioned ordered list;
• saving the identified last key value for each logical shard in a meta-store for the physical shard to which the logical shard is mapped;
• and
• partially re-indexing the plurality of shards of the one or more clusters based on addition of a node to the plurality of node in the one or more clusters, wherein partially re-indexing the plurality of shards comprises selecting a set of largest shards in the one or more clusters based on an index size of each shard, redirecting traffic for each shard of the selected set of shards to a different cluster of the one or more clusters while the plurality of shards are being partially reindexed, and re-directing traffic back to each shard of the selected set of shards upon completion of partially reindexing the plurality of shards.

A non-transitory, computer-readable medium comprising a set of instructions stored therein which, when executed by a processor, causes the processor to distribute data across a plurality of storage shards by:

• generating a file key for each file of a plurality of files stored in a plurality of physical shards, each physical shard maintained by a node of a plurality of nodes in one or more clusters, wherein the file key comprises a hash of an enterprise identifier (enterprise ID) for an entity to which the creator of the file is a member, a hash of a folder identifier (folder ID) for a location in which the file is stored, and a hash of a file identifier (file ID) uniquely identifying the file;
• sorting the generated file keys for each file of the plurality of files into an ordered list;
• logically partitioning the ordered list into a plurality of logical shards;
• mapping each logical shard of the plurality of logical shards to one of the plurality of physical shards;
• identifying a last key value for each logical shard in the partitioned ordered list;
• saving the identified last key value for each logical shard in a meta-store for the physical shard to which the logical shard is mapped;
• and
• partially re-indexing the plurality of shards of the one or more clusters based on addition of a node to the plurality of node in the one or more clusters, wherein partially re-indexing the plurality of shards comprises selecting a set of largest shards in the one or more clusters based on an index size of each shard, redirecting traffic for each shard of the selected set of shards to a different cluster of the one or more clusters while the plurality of shards are being partially reindexed, and re-directing traffic back to each shard of the selected set of shards upon completion of partially reindexing the plurality of shards.