Planning of data segment merge for distributed storage system

A computer program product for planning data segment merging within a distributed storage system including a plurality of data node servers and a plurality of indices, each data node server of the plurality of data node servers having a plurality of data segments, the plurality of data node servers in communication with:

• a client application through an interface, a disk, a data merge engine, and a computer comprising at least one processor, one or more memories, one or more computer readable storage media, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by the computer to perform a method comprising:
• collecting a record of usage statistics of the distributed storage system;
• analyzing a historical record of usage statistics collected to determine at least one usage pattern of the distributed storage system;
• estimating a workload of each of the plurality of data node servers and a length of time for a segment merge;
• determining a plan for merging;
• and
• sending the plan to the plurality of data node servers for implementation;
• wherein the plan for merging has a first stage of merging of data segments on one data node server and a second stage of merging of data segments across a set of the plurality of data node servers, wherein a set of data segments is distributed across the set of the plurality of data node servers and is included in an index, the first stage including:
• sorting, by the computer, data size of data segments in an index on the one data node server in ascending order;
• choosing, by the computer, a data segment with a minimum data size and data segment file number greater than one;
• adding, by the computer, the data segments of the index into the plan for merging, setting the merge rate, and estimating, by the computer, time length of merge for the data segments into a single data segment for the index;
• and
• executing, by the computer, the merge of the data segments into a single data segment for the index;
• and
• the second stage including:
• identifying an index including a set of data segments, wherein the set of data segments is distributed across the set of data node servers;
• sorting, for each data node server of the set of data node servers, all data segments of the set of data segments stored on the each data node server by data size;
• choosing a data node server of the set of data node servers with a minimum data size;
• setting a merge rate for the data node server chosen;
• and
• executing a merge of each data segment stored on the data node server chosen included in the index into a single data segment of the data node server chosen.

A computer system for planning data segment merging within a distributed storage system including a plurality of data node servers and a plurality of indices, each data node server of the plurality of data node servers having a plurality of data segments, the plurality of data node servers in communication with:

• a client application through an interface, a disk, a data merge engine, the plurality of data node servers comprising a computer comprising at least one processor, one or more memories, one or more computer readable storage media having program instructions executable by the computer to perform operations comprising:
• collecting a record of usage statistics of the distributed storage system;
• analyzing a historical record of usage statistics collected to determine at least one usage pattern of the distributed storage system;
• estimating a workload of each of the plurality of data node servers and a length of time for a segment merge;
• determining a plan for merging;
• and
• sending the plan to the plurality of data node servers for implementation;
• wherein the plan for merging has a first stage of merging of data segments on one data node server and a second stage of merging of data segments across a set of the plurality of data node servers, wherein a set of data segments is distributed across the set of the plurality of data node servers and is included in an index from the plurality of indices, the first stage including:
• sorting, by the computer, data size of data segments in an index on the one data node server in ascending order;
• choosing, by the computer, a data segment with a minimum data size and data segment file number greater than one;
• adding, by the computer, the data segments of the index into the plan for merging, setting the merge rate, and estimating, by the computer, time length of merge for the data segments into a single data segment for the index:
• and
• executing, by the computer, the merge of the data segments into a single data segment for the index;
• and
• the second stage including:
• identifying an index including a set of data segments, wherein the index is included in the plurality of indices, and wherein the set of data segments is distributed across the set of data node servers;
• sorting, for each data node server of the set of data node servers, all data segments of the set of data segments stored on the each data node server by data size;
• choosing a data node server of the set of data node servers with a minimum data size;
• setting a merge rate for the data node server chosen;
• and
• executing a merge of each data segment stored on the data node server chosen included in the index into a single data segment of the data node server chosen.