Serverless execution of code using cluster resources

A method for cluster resource allocation, comprising:

• receiving a process and input data;
• determining, using a processor, an estimate for resources required for the process to process the input data, comprising:
• determining a parallelism level for one or more steps of the process, the parallelism level including a quantity of data per cluster worker, a number of cluster workers, or a quantity of data per processor;
• determining an implementation type for one or more steps of the process, the implementation type including a sort algorithm a join algorithm or a filter algorithm;
• and
• determining, based on historical usage data, the estimate for resources required for the process using the parallelism level and the implementation type;
• determining existing available resources in a cluster for running the process;
• determining whether the existing available resources are sufficient for running the process based at least in part on the estimate for resources required;
• in response to determining that the existing available resources are not sufficient for running the process, indicating to add new resources to a cluster;
• determining an allocated share of resources in the cluster for running the process;
• causing execution of the process using the allocated share of resources, comprising:
• providing an indication of the share of resources, the process, and the input data to a cluster driver;
• and
• providing an indication to the cluster driver to execute the process of the input data using the share of resources.

A computer program product for cluster resource allocation, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

• receiving a process and input data;
• determining an estimate for resources required for the process to process the input data, comprising:
• determining a parallelism level for one or more steps of the process, the parallelism level including a quantity of data per cluster worker, a number of cluster workers, or a quantity of data per processor;
• determining an implementation type for one or more steps of the process, the implementation type including a sort algorithm a join algorithm or a filter algorithm;
• and
• determining, based on historical usage data, the estimate for resources required for the process using the parallelism level and the implementation type;
• determining existing available resources in a cluster for running the process;
• determining whether the existing available resources are sufficient for running the process based at least in part on the estimate for resources required;
• in response to determining that the existing available resources are not sufficient for running the process, indicating to add new resources to a cluster;
• determining an allocated share of resources in the cluster for running the process;
• causing execution of the process using the allocated share of resources, comprising:
• providing an indication of the share of resources, the process, and the input data to a cluster driver;
• and
• providing an indication to the cluster driver to execute the process of the input data using the share of resources.