Systems and methods for dynamic partitioning in distributed environments

A system for dynamic partitioning in distributed computing environments, the system including:

• a data storage device that stores instructions for dynamic partitioning in distributed computing environments;
• and
• a processor configured to execute the instructions to perform a method including:
• receiving a first data set and a second data set;
• mapping the first data set into a first set of key-value pairs;
• mapping the second data set into a second set of key-value pairs;
• estimating, using a sketch, a frequency count for each key based on the first set of key-value pairs and the second set of key-value pairs;
• retrieving, by the processor from a master node, a global frequency count for each key of the key-value pairs;
• determining, by the processor, an updated frequency count for each key based on the global frequency count for each key and the estimated frequency count for each key;
• determining, by the processor, whether the estimated frequency count for each key is greater than or equal to a predetermined threshold based on the updated frequency count for each key;
• and
• partitioning, by the processor, the key when the estimated frequency count for the key is greater than or equal to the predetermined threshold.

A non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for dynamic partitioning in distributed computing environments, the method including:

• receiving a first data set and a second data set;
• mapping the first data set into a first set of key-value pairs;
• mapping the second data set into a second set of key-value pairs;
• estimating, using a sketch, a frequency count for each key based on the first set of key-value pairs and the second set of key-value pairs;
• retrieving, by the processor from a master node, a global frequency count for each key of the key-value pairs;
• determining, by the processor, an updated frequency count for each key based on the global frequency count for each key and the estimated frequency count for each key;
• determining, by the processor, whether the estimated frequency count for each key is greater than or equal to a predetermined threshold based on the updated frequency count for each key;
• and
• partitioning, by the processor, the key when the estimated frequency count for the key is greater than or equal to the predetermined threshold.