Large distributed database clustering systems and methods

A computer-implemented method for managing a distributed database, implementing a sharded data architecture and replica set configuration of computer systems hosting the distributed database, the method comprising:

• organizing, by a computer system, a plurality of database partitions and assigning a key value or key pattern to new data added to the database greater than previously used key values or patterns;
• hosting, by a shard server, one or more of the plurality of database partitions, wherein the shard server comprises a replica set;
• replicating, by the replica set, the one or more of the plurality of database partitions from a primary system to at least two secondary systems that comprise the replica set;
• cloning, automatically, the one or more of the plurality of database partitions by adding a new computer system to the replica set;
• splitting, the one more of the plurality of database partitions into at least one additional partition during the act of cloning, wherein the act of splitting includes filtering at least some data from the one or more of the plurality of database partitions into the at least one additional partition;
• monitoring, by the computer system, a distributed database including a plurality of database partitions, wherein at least one of the plurality of database partitions includes a contiguous range of data from the database;
• detecting, by the computer system, a partition size of the at least one of the plurality of database partitions exceeds a size threshold;
• splitting, by the computer system, the at least one of the plurality of database partitions into at least a first and a second partition;
• and
• controlling, by the computer system, a non-uniform distribution of data within the first and the second partition based on unique values for database keys associated with the data in the at least one of the plurality of database partitions, wherein controlling the non-uniform distribution includes minimizing any data distributed to the second partition based on a maximum value for the database keys associated with the data in the at least one of the plurality of database partitions;
• and
• routing, by the computer system, new data that would have been directed to the at least one partition prior to the split to the second partition.

A system for managing a large distributed database, the system comprising:

• a distributed database;
• at least one processor operatively connected to a memory, the at least one processor configured to execute a plurality of system components from the memory;
• a configuration component, executed by the at least one processor, configured to organize the distributed database into a plurality of database partitions, wherein the configuration component provides data routing information for accepting and routing client requests to the plurality of database partitions;
• a shard server configured to host one or more of the plurality of database partitions, wherein the shard server further comprises a replica set,
• wherein the replica set comprises:
• a primary node and at least two secondary nodes for replication of a primary copy of the one or more of the plurality of the database partitions to secondary copies of the one or more partitions hosted on the secondary nodes;
• and wherein the replica set is configured to replicate data operations within the replica set based on an eventually consistent model;
• a recovery component, executed by the at least one processor, configured to:
• automatically establish a new primary node in the distributed database system in response to a replication failure, wherein automatically establishing includes executing functions to establish one of the plurality of secondary nodes as the new primary node, and delete replicated write operations from the new primary node received from the former primary node to establish a point of consistency within the database between the new primary node and the remaining secondary nodes as a new current state of the database;
• and
• a partition component configured to control a non-uniform distribution of data during splitting of at least one of the plurality of database partitions into a first and a second partition based, at least in part, on unique values for database keys associated with the data in the at least one of the plurality of database partitions, wherein controlling the non-uniform distribution includes maximizing any data distributed to the first partition and minimizing any data distributed to the second partition based on a maximum value for the database keys associated with the data in the at least one of the plurality of database partitions;
• and
• a routing component configured to route new data based on having a database key value that exceeds the maximum value to the second partition, wherein the new data is assigned increasing key values relative to existing data.

A computer implemented method for managing a large distributed database, the method comprising:

• organizing, by a computer system, a distributed database into a plurality of database partitions, wherein organizing the distributed database includes defining data routing information for accepting and routing client requests to the plurality of non-uniform database partitions;
• hosting, by the computer system, one or more of the plurality of database partitions;
• replicating, by the computer system, a respective one of the one or more of the plurality of database partitions between a primary node and at least two secondary nodes;
• automatically recovering a primary node role in the distributed database system in response to a failure of the primary node, wherein the act of automatically recovering includes establishing one of the plurality of secondary nodes as a new primary node, and wherein establishing the one of the plurality of secondary nodes as the new primary node includes deleting replicated write operations by the new primary node received from the primary node to establish a point of consistency within the database between the new primary node and the remaining secondary nodes as a new current state of the database;
• wherein the primary node and at least two secondary nodes host a primary copy of the one or more of the plurality of the database partitions at the primary node, and the act of replicating includes replicating the primary copy to secondary copies of the one or more partitions hosted on the secondary nodes;
• and
• controlling, by the computer system, a non-uniform distribution of data during an act of splitting at least one of the plurality of database partitions into a first and a second partition based on unique values for database keys associated with the data in the at least one of the plurality of database partitions, wherein controlling the non-uniform distribution includes maximizing any data distributed to the first partition and minimizing any data distributed to the second partition based on a maximum value for the database keys associated with the data in the at least one of the plurality of database partitions;
• routing, by the computer system, new data based on having a database key value that exceeds the maximum value to the second partition, wherein the new data is assigned increasing key values relative to existing data.