Autoscaling for streaming workloads

A method, comprising:

• obtaining input data from one or more data sources;
• determining, by a data layer comprising one or more clusters of virtual machines, a set of the input data to process;
• obtaining a current measure of a number of queued tasks and/or a task-processing capacity, wherein the current measure of the number of queued tasks and/or a task-processing capacity are obtained from a data structure that is exposed to the data layer and that is isolated from a control layer, wherein the control layer comprises a cluster manager that manages a set of the one or more clusters, wherein each cluster of the set of one or more clusters comprises a driver node and a set of worker nodes, the driver node and set of worker nodes at the data layer;
• determining a set of tasks to process based at least in part on the set of the input data to process, wherein determining the set of tasks further comprises determining whether the input data is bounded, wherein bounded data is batched data and unbounded data is streaming data, and in response to determining that the data is unbounded:
• determining a microbatch for the data, and
• determining the set of tasks to be performed in order to process the microbatch of data;
• and
• automatically scaling the task-processing capacity based at least in part on (i) the current measure of the number of queued tasks, a measure of the task-processing capacity, and/or a measure of utilization of the task-processing capacity, and (ii) the set of tasks to process.

A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

• obtaining input data from one or more data sources;
• determining, by a data layer comprising one or more clusters of virtual machines, a set of the input data to process;
• obtaining a current measure of a number of queued tasks and/or a task-processing capacity, wherein the current measure of the number of queued tasks and/or a task-processing capacity are obtained from a data structure that is exposed to the data layer and that is isolated from a control layer, wherein the control layer comprises a cluster manager that manages a set of the one or more clusters, wherein each cluster of the set of one or more clusters comprises a driver node and a set of worker nodes, the driver node and set of worker nodes at the data layer;
• determining a set of tasks to process based at least in part on the set of the input data to process, wherein determining the set of tasks further comprises determining whether the input data is bounded, wherein bounded data is batched data and unbounded data is streaming data, and in response to determining that the data is unbounded:
• determining a microbatch for the data, and
• determining the set of tasks to be performed in order to process the microbatch of data;
• and
• automatically scaling the task-processing capacity based at least in part on (i) the current measure of the number of queued tasks, a measure of the task-processing capacity, and/or a measure of utilization of the task-processing capacity, and/or the task-processing capacity, and (ii) the set of tasks to process.