Parallel processing on data processing servers through even division of data records

A data processing apparatus, implemented on a client in a distributed data processing system;

• wherein the distributed data processing system comprises:
• the client, a task scheduler, a plurality of data processing servers and a plurality of storage servers;
• the plurality of storage servers store a plurality of data regions, each of which contains a plurality of data records, the data processing apparatus comprising:
• a processing instruction receiving module, configured to receive a data processing instruction for to-be-processed data;
• an active data determining module, configured to determine active data regions according to the data processing instruction;
• wherein each of the active data regions is a data region of the to-be-processed data;
• a distribution information obtaining module, configured to obtain data distribution information of data records in each active data region of the active data regions;
• wherein the data distribution information is information obtained by calculating a number of data records in each active data region through statistics according to a preset statistical rule;
• a data range dividing module, configured to divide a data range of each of the active data regions into at least one sub-data range according to the data distribution information;
• a task generating module, configured to generate data processing tasks for the active data regions according to the at least one sub-data range obtained from the division;
• wherein one data processing task corresponds to one sub-data range of one active data region;
• a processing request sending module, configured to send a data processing request for the data processing tasks to the task scheduler, such that the task scheduler determines for each of the data processing tasks a data processing server of the plurality of data processing servers to execute it, according to the data processing request;
• wherein the task scheduler, after determining the plurality of data processing servers, sends the data processing tasks to the determined plurality of data processing servers for execution, and receives processing results of performing the data processing tasks on allocated sub-data ranges from the plurality of data processing servers;
• a processing result receiving module, configured to receive a processing result of the to-be-processed data fed back by the task scheduler;
• wherein a storage server of the plurality of storage servers comprises:
• a data sorting submodule, an interval dividing submodule, a data statistics submodule and a distribution information obtaining submodule;
• wherein
• the data sorting submodule is configured to sort data records in a data region Ri by a preset identifier, wherein i is an integer number, i≥1, and Ri represents an ith data region in the active data regions;
• the interval dividing submodule is configured to divide the data region Ri into data intervals according to a sorted order of the sorted data records in the data region Ri;
• the data statistics submodule is configured to obtain a number of data records in each of the data intervals through statistics;
• the distribution information obtaining submodule is configured to obtain the data distribution information of the data records in the data region Ri according to the number of data records in each of the data intervals obtained through statistics;
• wherein the distribution information obtaining submodule comprises:
• an adjustment condition determining unit and a first interval adjustment unit, wherein
• the adjustment condition determining unit is configured to determine whether the number of data records in each of the data intervals in the data region Ri satisfies a preset data interval adjustment condition;
• wherein the adjustment condition determining unit comprises:
• an interval variance calculating subunit, an interval variance determining subunit and a processing condition determining subunit;
• wherein
• the interval variance calculating subunit is configured to calculate a variance of the number of data records in each of the data intervals in the data region Ri obtained through statistics;
• the interval variance determining subunit is configured to determine whether the variance obtained from the calculation is greater than a preset variance threshold;
• and
• the processing condition determining subunit is configured to, when the variance is greater than the preset variance threshold, determine that the number of data records in each of the data intervals in the data region Ri obtained through statistics satisfies the preset data interval processing condition;
• the first interval adjustment unit is configured to, when the variance is greater than the preset variance threshold, adjust each of the data intervals in the data region Ri according to a preset interval adjustment rule, and obtain the data distribution information of data records in the data region Ri according to a number of data records in each of the adjusted data intervals;
• wherein the preset interval adjustment rule comprises:
• obtaining, in the data region Ri, data intervals having a number of data records greater than a first preset number threshold Th1, and obtaining, in the data region Ri, data intervals having a number of data records smaller than a second preset number threshold Th2, wherein Th1≥Th2;
• according to a preset data interval split rule, splitting each of the data intervals in the data region Ri having a number of data records greater than Th1 into a plurality of sub-data intervals;
• and
• according to a preset data interval merge rule, merging data intervals in the data region Ri having a number of data records smaller than Th2.

A client device in a distributed data processing system, wherein the distributed data processing system further comprises:

• a task scheduler, a plurality of data processing servers and a plurality of storage servers;
• the plurality of storage servers store a plurality of data regions, each of which contains a plurality of data records;
• the client device comprising:
• a housing, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed in a space enclosed by the housing, and the processor and the memory are disposed on the circuit board;
• the power supply circuit is configured to supply power to circuits or components of the client device;
• the memory is configured to store executable program code;
• the processor executes a program corresponding to the executable program code by reading the executable program code stored in the memory, to perform the following steps:
• receiving a data processing instruction for to-be-processed data;
• determining active data regions according to the data processing instruction;
• wherein each of the active data regions is a data region of the to-be-processed data;
• obtaining data distribution information of data records in each active data region of the active data regions;
• wherein the data distribution information is information obtained by calculating a number of data records in each active data region through statistics according to a preset statistical rule;
• dividing a data range of each of the active data regions into at least one sub-data range according to the data distribution information;
• generating data processing tasks for the active data regions according to the at least one sub-data range obtained from the division;
• wherein one data processing task corresponds to one sub-data range of one active data region;
• sending a data processing request for the data processing tasks to the task scheduler, such that the task scheduler determines for each of the data processing tasks a data processing server of the plurality of data processing servers to execute it, according to the data processing request, wherein the task scheduler, after determining the plurality of data processing servers, sends the data processing tasks to the determined plurality of data processing servers for execution, and receives processing results of performing the data processing tasks on allocated sub-data ranges from the plurality of data processing servers;
• receiving a processing result of the to-be-processed data fed back by the task scheduler;
• wherein a storage server of the plurality of storage servers obtains the data distribution information of data records in a data region Ri of the active data regions through statistics by performing these following steps, wherein i is an integer number, i≥1, and Ri represents an ith data region in the active data regions:
• sorting, by the storage server of the plurality of storage servers, the data records in the data region Ri by a preset identifier;
• dividing the data region Ri into data intervals according to a sorted order of the sorted data records in the data region Ri;
• obtaining a number of data records in each of the data intervals through statistics;
• wherein obtaining the data distribution information of the data records in the data region Ri according to the number of data records in each of the data intervals obtained through statistics, comprises:
• determining whether the number of data records in each of the data intervals in the data region Ri satisfies a preset data interval adjustment condition;
• wherein the determining, by the storage server of the plurality of storage servers, whether the data region Ri satisfies the preset data interval adjustment condition comprises:
• calculating a variance of the number of data records in each of the data intervals in the data region Ri obtained through statistics;
• determining whether the variance obtained from the calculation is greater than a preset variance threshold;
• if the variance is greater than the preset variance threshold, determining that the number of data records in each of the data intervals in the data region Ri obtained through statistics satisfies the preset data interval adjustment condition;
• when the variance is greater than the preset variance threshold, adjusting each of the data intervals in the data region Ri according to a preset interval adjustment rule, and obtaining the data distribution information of data records in the data region Ri according to a number of data records in each of the adjusted data intervals;
• wherein the preset interval adjustment rule comprises:
• obtaining, in the data region Ri, data intervals having a number of data records greater than a first preset number threshold Th1, and obtaining, in the data region Ri, data intervals having a number of data records smaller than a second preset number threshold Th2, wherein Th1≥Th2;
• according to a preset data interval split rule, splitting each of the data intervals in the data region Ri having a number of data records greater than Th1 into a plurality of sub-data intervals;
• and
• according to a preset data interval merge rule, merging data intervals in the data region Ri having a number of data records smaller than Th2.

A non-transitory storage medium configured to store an application program which, when executed, performs operations of:

• receiving a data processing instruction for to-be-processed data;
• determining active data regions according to the data processing instruction;
• wherein each of the active data regions is a data region of the to-be-processed data;
• obtaining data distribution information of data records in each active data region of the active data regions;
• wherein the data distribution information is information obtained by calculating a number of data records in each active data region through statistics according to a preset statistical rule;
• dividing a data range of each of the active data regions into at least one sub-data range according to the data distribution information;
• generating data processing tasks for the active data regions according to the at least one sub-data range obtained from the division;
• wherein one data processing task corresponds to one sub-data range of one active data region;
• sending a data processing request for the data processing tasks to a task scheduler, such that the task scheduler determines for each of the data processing tasks a data processing server to execute it, according to the data processing request, wherein the task scheduler, after determining data processing servers, sends the data processing tasks to the determined data processing servers for execution, and receives processing results of performing the data processing tasks on allocated sub-data ranges from the data processing servers;
• receiving a processing result of the to-be-processed data fed back by the task scheduler;
• wherein a storage server obtains the data distribution information of data records in a data region Ri of the active data regions through statistics by performing these following steps, wherein i is an integer number, i≥1, and Ri represents an ith data region in the active data regions:
• sorting, by the storage server, the data records in the data region Ri by a preset identifier;
• dividing the data region Ri into data intervals according to a sorted order of the sorted data records in the data region Ri;
• obtaining a number of data records in each of the data intervals through statistics;
• wherein obtaining the data distribution information of the data records in the data region Ri according to the number of data records in each of the data intervals obtained through statistics, comprises:
• determining whether the number of data records in each of the data intervals in the data region Ri satisfies a preset data interval adjustment condition;
• wherein the determining, by the storage server, whether the data region Ri satisfies the preset data interval adjustment condition comprises:
• calculating a variance of the number of data records in each of the data intervals in the data region Ri obtained through statistics;
• determining whether the variance obtained from the calculation is greater than a preset variance threshold;
• if the variance is greater than the preset variance threshold, determining that the number of data records in each of the data intervals in the data region Ri obtained through statistics satisfies the preset data interval adjustment condition;
• when the variance is greater than the preset variance threshold, adjusting each of the data intervals in the data region Ri according to a preset interval adjustment rule, and obtaining the data distribution information of data records in the data region Ri according to a number of data records in each of the adjusted data intervals;
• wherein the preset interval adjustment rule comprises:
• obtaining, in the data region Ri, data intervals having a number of data records greater than a first preset number threshold Th1, and obtaining, in the data region Ri, data intervals having a number of data records smaller than a second preset number threshold Th2, wherein Th1≥Th2;
• according to a preset data interval split rule, splitting each of the data intervals in the data region Ri having a number of data records greater than Th1 into a plurality of sub-data intervals;
• and
• according to a preset data interval merge rule, merging data intervals in the data region Ri having a number of data records smaller than Th2.