State rebalancing in structured streaming

A computer-implemented method comprising:

• executing, using a set of executors, a set of partition tasks on at least a first batch of data, wherein a first partition task from the set of partition tasks is assigned to execute on a first executor from the set of executors;
• computing, based on placement information indicating locations of the set of partition tasks on the set of executors, preferred locations of the set of partition tasks for at least a second batch of data that was received subsequent to the first batch of data, the preferred locations indicating at least reassignment of the first partition task from the first executor to a second executor from the set of executors;
• and
• executing, using the set of executors, the set of partition tasks on at least the second batch of data based on the preferred locations, wherein the first partition task is assigned to the second executor rather than the first executor.

A non-transitory computer readable medium comprising stored instructions, the stored instructions when executed by at least one computer processor of one or more computing devices, cause the one or more computing devices to:

• execute, using a set of executors, a set of partition tasks on at least a first batch of data, wherein a first partition task from the set of partition tasks is assigned to execute on a first executor from the set of executors;
• compute, based on placement information indicating locations of the set of partition tasks on the set of executors, preferred locations of the set of partition tasks for at least a second batch of data that was received subsequent to the first batch of data, the preferred locations indicating at least reassignment of the first partition task from the first executor to a second executor from the set of executors;
• and
• execute, using the set of executors, the set of partition tasks on at least the second batch of data based on the preferred locations, wherein the first partition task is assigned to the second executor rather than the first executor.

A computer system, comprising:

• at least one computer processor;
• and
• a non-transitory computer-readable storage medium comprising instructions that when executed by the at least one computer processor, cause the computer system to:
• execute, using a set of executors, a set of partition tasks on at least a first batch of data, wherein a first partition task from the set of partition tasks is assigned to execute on a first executor from the set of executors;
• compute, based on placement information indicating locations of the set of partition tasks on the set of executors, preferred locations of the set of partition tasks for at least a second batch of data that was received subsequent to the first batch of data, the preferred locations indicating at least reassignment of the first partition task from the first executor to a second executor from the set of executors;
• and
• execute, using the set of executors, the set of partition tasks on at least the second batch of data based on the preferred locations, wherein the first partition task is assigned to the second executor rather than the first executor.