System for performing a redistribute transparently in a multi-node system
AssigneeInternational Business Machines Corporation
InventorsRonen Grosman, Keriley Kay Romanufa, Robin D. Van Boeschoten
Life of this patent
Estimating the term…
Abstract
Performing a redistribute of data in a database system including a plurality of nodes is disclosed. The data includes a plurality of partitions distributed between the plurality of nodes. At least one new node is being added. At least one partition of the plurality of partitions is selected to be moved from the plurality of nodes only to the at least one new node. The at least one partition is moved only to the at least one new node. At least one partition is removed from the plurality of nodes.
Claims19
A computer system for performing a redistribution of data in a database system including a plurality of preexisting nodes, at least one new node being added, the computer system comprising:
- a processing element for selecting, in response to the at least one new node being added to the database system, at least one partition of a plurality of partitions distributed between the plurality of preexisting nodes such that each preexisting node has one or more partitions, wherein
- the partitions are substructures for the preexisting nodes and the rows of data of the database have been hashed into the partitions such that the number of partitions is greater than the number of preexisting nodes,
- each of the plurality of nodes and each of the at least one new node include a portion of the data and the processing element for selecting includes a processing element for choosing the at least one partition to minimize a difference between the portion of the data in each of the plurality of preexisting nodes and each of the at least one new node,
- the processing element for selecting uses a global ownership table to distinguish between the preexisting nodes and the new nodes in the database system, and
- each of the at least one partition corresponds to an index;
- a processing element for moving the at least one selected partition only to the at least one new node without moving one or more partitions from one of the preexisting nodes to a different one of the preexisting nodes, the processing element for moving the at least one selected partition including
- a processing element for transparently accounting for the index of each of the at least one selected partition, including and element for building a new index for each of the at least one selected partition moved to the new node and a processing element for marking the index for each of the at least one selected partition as deleted on the corresponding preexisting node, such that the at least one selected partition is treated as if deleted from the corresponding preexisting node and does not need to actually be deleted immediately in response to the moving;
- and
- a processing element for removing the at least one selected partition from the plurality of preexisting nodes.
A computer program product comprising a computer-readable storage medium including a program implemented by a computer and performing a redistribution of data in a database system including a plurality of preexisting nodes, at least one new node being added, the program including instructions for:
- in response to the at least one new node being added to the database system selecting, at least one partition of a plurality of partitions distributed between the plurality of preexisting nodes such that each preexisting node has one or more partitions, wherein
- the partitions are substructures for the preexisting nodes and the rows of data of the database have been hashed into the partitions such that the number of partitions is greater than the number of preexisting nodes,
- each of the plurality of nodes and each of the at least one new node include a portion of the data and the selecting includes choosing the at least one partition to minimize a difference between the portion of the data in each of the plurality of preexisting nodes and each of the at least one new node,
- the selecting uses a global ownership table to distinguish between the preexisting nodes and the new nodes in the database system, and
- each of the at least one partition corresponds to an index;
- moving the at least one selected partition only to the at least one new node without moving one or more partitions from one of the preexisting nodes to a different one of the preexisting nodes, the moving the at least one selected partition including
- transparently accounting for the index of each of the at least one selected partition, including and building a new index for each of the at least one selected partition moved to the new node and marking the index for each of the at least one selected partition as deleted on the corresponding preexisting node, such that the at least one selected partition is treated as if deleted from the corresponding preexisting node and does not need to actually be deleted immediately in response to the moving;
- and
- removing the at least one selected partition from the plurality of preexisting nodes.
A computer program product comprising a computer-readable storage medium including a program implemented by a computer and performing a redistribution of data in a database system including a plurality of preexisting nodes, at least one new node being added, the program including instructions for:
- in response to the at least one new node being added to the database system, selecting at least one partition of a plurality of partitions distributed between the plurality of preexisting nodes such that each preexisting node has one or more partitions, wherein the partitions are substructures for the preexisting nodes and the rows of data of the database have been hashed into the partitions such that the number of partitions is greater than the number of preexisting nodes, wherein
- each of the plurality of nodes and each of the at least one new node include a portion of the data and the selection includes choosing the at least one partition to minimize a difference between the portion of the data in each of the plurality of preexisting nodes and each of the at least one new node, wherein
- the selection is performed using a global ownership table to distinguish between the preexisting nodes and the new nodes in the database system, and wherein
- each of the at least one partition corresponds to an index;
- moving the at least one selected partition only to the at least one new node without moving one or more partitions from one of the preexisting nodes to a different one of the preexisting nodes, the moving the at least one selected partition including
- copying the at least one selected partition to the at least one new node;
- building a new index on each of the at least one new node for each of the at least one selected partition;
- maintaining an activity log for each of the at least one selected partition on the plurality of preexisting nodes, including recording operations in the activity log, the operations being for the data in the at least one selected partition being copied;
- suspending access to the data, wherein the operations occurring while the new index is built are recorded in the activity log during the suspending of access to the data;
- and
- applying the activity log for each of the at least one selected partition on each of the at least one new node such that the operations recorded in the activity log are applied to the at least one selected partition and changes occurring while the new index was built are accounted for;
- and
- marking the index for each of the at least one partition as deleted;
- and
- removing the at least one selected partition from the plurality of preexisting nodes.
Dependencies are inferred from the claim text.