Elastic sharding of data in a multi-tenant cloud

A system comprising:

• one or more hardware processors;
• and
• a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
• receiving, via a computer network, first primary data from a first tenant;
• receiving first auxiliary data relating to the first primary data from the first tenant;
• creating, using one or more hardware processors, a first index for the first primary data and the first auxiliary data from the first tenant;
• storing, in a memory, the first index as a first shardlet;
• bundling the first shardlet with one or more other shardlets for the first tenant in a first shard group for the first tenant, causing the first shard group to contain a plurality of shardlets for the first tenant;
• receiving second primary data from a second tenant in the computer network;
• receiving second auxiliary data relating to the second primary data from the second tenant;
• creating, using the one or more hardware processors, a second index for the second primary data and the second auxiliary data from the second tenant;
• storing, in the memory, the second index as a second shardlet;
• bundling the second shardlet with one or more other shardlets for the second tenant in a second shard group, causing the second shard group to contain a plurality of shardlets for the second tenant;
• packing the first shard group and second shard group in a first shard, causing the first shard to contain the plurality of shardlets for the first tenant and the plurality of shardlets for the second tenant;
• storing the first shard in a first instance of a distributed database, the distributed database comprising a plurality of instances, each instance operating on a different logical or physical device;
• distributing the first shard to a first search core;
• distributing a second shard to a second search core;
• receiving a first search request;
• determining that the first search request involves data located at an index included as a shardlet in a shard group in the first shard;
• and
• in response to the determining, forwarding the first search request to the first search core in lieu of the second search core.

A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines, cause the one or more machines to perform operations comprising:

• receiving, via a computer network, first primary data from a first tenant;
• receiving first auxiliary data relating to the first primary data from the first tenant;
• creating, using one or more hardware processors, a first index for the first primary data and the first auxiliary data from the first tenant;
• storing, in a memory, the first index as a first shardlet;
• bundling the first shardlet with one or more other shardlets for the first tenant in a first shard group for the first tenant, causing the first shard group to contain a plurality of shardlets for the first tenant;
• receiving second primary data from a second tenant in the computer network;
• receiving second auxiliary data relating to the second primary data from the second tenant;
• creating, using the one or more hardware processors, a second index for the second primary data and the second auxiliary data from the second tenant;
• storing, in the memory, the second index as a second shardlet;
• bundling the second shardlet with one or more other shardlets for the second tenant in a second shard group, causing the second shard group to contain a plurality of shardlets for the second tenant;
• packing the first shard group and second shard group in a first shard, causing the first shard to contain the plurality of shardlets for the first tenant and the plurality of shardlets for the second tenant;
• storing the first shard in a first instance of a distributed database, the distributed database comprising a plurality of instances, each instance operating on a different logical or physical device;
• distributing the first shard to a first search core;
• distributing a second shard to a second search core;
• receiving a first search request;
• determining that the first search request involves data located at an index included as a shardlet in a shard group in the first shard;
• and
• in response to the determining, forwarding the first search request to the first search core in lieu of the second search core.