Agile hostpool allocator

A computer-implemented method to programmatically allocate a processing element of a stream application to a compute node, based on hostpools and a hierarchy of constraints, the computer-implemented method comprising:

• providing a plurality of processing elements of the stream application, the plurality of processing elements including the processing element and pertaining to a requested compute job, wherein each of the plurality of processing elements includes one or more operators fused into a single process uniquely identifiable via a process identifier and having an associated memory space, wherein operators of different processing elements communicate via an inter-nodal communication protocol, wherein operators of the same processing element communicate via an intra-nodal communication protocol;
• providing an operator graph defining an execution path through the plurality of processing elements, the execution path specifying that at least one of the processing elements receives data from at least one upstream processing element and transmits data to at least one downstream processing element, wherein the plurality of processing elements are assignable to a plurality of compute nodes in a distributed computing environment, the plurality of compute nodes including the compute node;
• providing a constraint tree defining the hierarchy of constraints, the hierarchy of constraints including, for each of a plurality of levels of the stream application, a respective constraint specific to the respective level, the plurality of levels including a host level, a hostpool level, an operator level, a processing-element level, and an element-relationship level;
• providing, using a plurality of hostpools, support for deferring assignment of processing elements until runtime, which provides a greater measure of flexibility in assigning processing elements to compute nodes than is afforded by assigning, prior to runtime, processing elements to compute nodes in an absence of the plurality of hostpools, wherein providing the support includes assigning, to the processing element, a hostpool selected from the plurality of hostpools, each of which has a distinct set of resource type requirements, wherein the hostpool will define a set of compute nodes that have been assigned to the plurality of processing elements, wherein the constraint tree specifies one or more resource type requirements of the hostpool at the hostpool level of the hierarchy;
• determining whether a number of compute nodes previously assigned to the hostpool is equal to a predetermined size of the hostpool, wherein the predetermined size is a maximum number of compute nodes that can be assigned to the hostpool;
• determining, during runtime and by operation of one or more computer processors, a plurality of candidate compute nodes in the distributed computing environment to dynamically assign to the hostpool, wherein the constraint tree specifies a respective one or more resource type requirements satisfied by each candidate compute node at the host level of the hierarchy;
• determining, according to a predefined final selection criterion, and based on a set of one or more hosting constraints associated with the processing element and specified by the constraint tree at the processing-element level of the hierarchy, a final compute node from the candidate compute nodes, the set of one or more hosting constraints specifying whether multiple processing elements in a specified group selected from the plurality of processing elements are permitted to execute on the same hosting compute node;
• assigning the processing element to the final compute node via the hostpool, including pinning the final compute node to the hostpool and slotting the final compute node to a specific index within the hostpool;
• and
• subsequent to assigning the processing element to the final compute node, generating a set of results by executing the stream application based on input comprising a set of data records, wherein generating the set of results includes incrementally generating each result based on a respective data record while the respective data record is in flight, wherein the result is output.