Short query prioritization for data processing service

A non-transitory computer-readable storage medium comprising stored instructions that, when executed by one or more computer processors, causes the one or more computer processors to:

• access a cluster including a driver node and one or more executor nodes each including one or more task engines, the cluster configured to schedule queries according to at least a first queue, a second queue, and a third queue, the first queue storing information on queries with a higher priority than queries assigned to the second queue, and the second queue storing information on queries with a higher priority than queries assigned to the third queue;
• receive a query and assigning the query to the first queue, the query divided into a set of tasks;
• allocate a number of task engines in a first set of executor nodes to process the set of tasks of the query;
• monitor a resource time for the query while the query is being executed at the first set of executor nodes;
• responsive to determining that the resource time for the query is equal to or above a threshold resource time, assign the query to the second queue;
• responsive to assigning the query to the second queue, execute the tasks of the query on a second set of executor nodes;
• responsive to determining that a second resource time for the query is equal to or above a second threshold resource time, assign the query to the third queue;
• and
• responsive to assigning the query to the third queue, execute the tasks of the query on a third set of executor nodes.

A computer system comprising:

• one or more computer processors;
• and
• one or more non-transitory computer readable storage mediums comprising stored instructions that, when executed by one or more computer processors, causes the one or more computer processors to:
• access a cluster including a driver node and one or more executor nodes each including one or more task engines, the cluster configured to schedule queries according to at least a first queue, a second queue, and a third queue, the first queue storing information on queries with a higher priority than queries assigned to the second queue, and the second queue storing information on queries with a higher priority than queries assigned to the third queue;
• receive a query and assigning the query to the first queue, the query divided into a set of tasks;
• allocate a number of task engines in a first set of executor nodes to process the set of tasks of the query;
• monitor a resource time for the query while the query is being executed at the first set of executor nodes;
• responsive to determining that the resource time for the query is equal to or above a threshold resource time, assign the query to the second queue;
• responsive to assigning the query to the second queue, execute the tasks of the query on a second set of executor nodes;
• responsive to determining that a second resource time for the query is equal to or above a second threshold resource time, assign the query to the third queue;
• and
• responsive to assigning the query to the third queue, execute the tasks of the query on a third set of executor nodes.