Adaptive data repartitioning and adaptive data replication

A method comprising:

• partitioning a data set stored in a distributed data processing system by a first partitioning key;
• processing a live workload comprising a plurality of data processing commands;
• while processing the live workload, maintaining statistical properties of the live workload;
• based on the statistical properties of the workload with respect to the data set, determining to replicate the data set in the distributed data processing system based on a second partitioning key;
• storing an additional copy of the data set in the distributed data processing system, wherein the additional copy of the data set is partitioned by the second partitioning key;
• wherein the method is performed by one or more computing devices.

A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause performance of:

• partitioning a data set stored in a distributed data processing system by a first partitioning key;
• processing a live workload comprising a plurality of data processing commands;
• while processing the live workload, maintaining statistical properties of the live workload;
• based on the statistical properties of the live workload with respect to the data set, determining to repartition the data set by a second partitioning key;
• repartitioning the data set in the distributed data processing system by the second partitioning key;
• wherein the method is performed by one or more computing devices.

A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause performance of:

• partitioning a data set stored in a distributed data processing system by a first partitioning key;
• processing a live workload comprising a plurality of data processing commands;
• while processing the live workload, maintaining statistical properties of the live workload;
• based on the statistical properties of the workload with respect to the data set, determining to replicate the data set in the distributed data processing system based on a second partitioning key;
• storing an additional copy of the data set in the distributed data processing system, wherein the additional copy of the data set is partitioned by the second partitioning key;
• wherein the method is performed by one or more computing devices.