Concurrent data processing in a distributed system

A method for allocating computing resources to vertices on a process node, the method comprising:

• on a process node, receiving information about a plurality of vertices assigned to the process node;
• receiving estimated resource usage information for at least one of the plurality of vertices assigned to the process node;
• allocating computing resources to a first assigned vertex;
• running the first assigned vertex;
• allocating computing resources to a second assigned vertex based on the received estimated resource usage information;
• running the second assigned vertex concurrently with the first assigned vertex;
• allocating computing resources to additional assigned vertices until either all computing resources have been allocated or computing resources are allocated to all additional assigned vertices;
• reserving an amount of memory for running vertices;
• reserving an amount of memory for running processes that are not vertices;
• and
• reserving an amount of free memory, wherein free memory is not available to be allocated to assigned vertices or available for running processes that are not vertices.

A vertex assignment scheduling system for scheduling vertices in a cluster, the system comprising:

• a computing device associated with a job manager having one or more process nodes and one or more computer-readable storage media;
• and
• a data store coupled with the job manager,
• wherein the job manager:
• assigns a plurality of vertices to the one or more process nodes that comprise a cluster,
• receives resource usage information for assigned vertices that have completed, and
• calculates, based on the received resource usage information for the completed vertices, estimated resource usage for the assigned vertices for which resource usage information has not been received;
• and wherein at least one of the one or more process nodes:
• transmits resource usage information to the job manager for each vertex assigned to the process node that completes,
• allocates computing resources based upon estimated resource usage transmitted to the process node by the job manager such that multiple vertices run concurrently on the process node, and
• allocates computing resources according to the position of assigned vertices in a queue on the process node, and wherein if the next vertex awaiting allocation of computing resources in the queue requires computing resources in excess of the available computing resources of the process node, the next vertex is flagged and is not allocated computing resources by the process node.