Self-tuning analytics system with observed execution optimization

A computer-implemented method comprising:

• receiving, by a resource manager of a distributed execution engine from a driver application, a request for execution resources for use in executing one or more tasks of a user application;
• selecting, by the resource manager, a type of executor from multiple candidate executor types based at least in part on one or more of:
• current execution data associated with the execution of tasks of the user application, or
• historic execution data associated with one or more other applications;
• wherein selecting, by the resource manager, the type of executor from multiple candidate executor types comprises:
• analyzing task execution history associated with the user application, or
• comparing a job or task profile of the user application to a job or task profile of another user application;
• and
• causing, by the resource manager, at least a first executor of the selected type to be launched for use in executing tasks of the user application.

A system comprising:

• a first one or more electronic devices to implement a data plane of an analytics service in a multi-tenant provider network, the data plane to host worker nodes to run executors for executing distributed user applications;
• and
• a second one or more electronic devices to implement a control plane of the analytics service in the multi-tenant provider network, the control plane including instructions that upon execution cause the control plane to:
• receive, from a driver application associated with a user application, a request for execution resources for use in executing one or more tasks of the user application;
• select a type of executor from multiple candidate executor types based at least in part on one or more of:
• current execution data associated with the execution of tasks of the user application, or
• historic execution data associated with one or more other applications executed by the data plane;
• wherein the instructions that upon execution cause the control plane to select the type of executor from multiple candidate executor types comprise instructions that upon execution cause the control plane to:
• analyze task execution history associated with the user application, or
• compare a job or task profile of the user application to a job or task profile of another user application;
• and
• cause at least a first executor of the selected type to be launched for use in executing tasks of the user application.