Optimizing resource allocation for distributed stream processing systems

A computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the stored program instructions comprising:

• program instructions to train a machine learning model to determine allocations of computing resources to processing elements of a stream processing job according to a specified objective;
• and
• program instructions to allocate a set of computing resources to the processing elements by:
• allocating to the processing elements a first subset of the set of computing resources based, at least in part, on a minimum resource requirement for the processing elements, and
• allocating to the processing elements a second subset of the set of computing resources based, at least in part, on an allocation determined using the trained machine learning model;
• wherein the allocating of the second subset is based on scoring of historical resource allocation information associated with the trained machine learning model and selecting CPU core adjustments of processing elements that result in a best score of the scoring.

A computer system comprising:

• a processor set;
• and
• a computer readable storage medium;
• wherein:
• the processor set is structured, located, connected and/or programmed to run program instructions stored on the computer readable storage medium;
• and
• the stored program instructions comprise:
• program instructions to train a machine learning model to determine allocations of computing resources to processing elements of a stream processing job according to a specified objective, and
• program instructions to allocate a set of computing resources to the processing elements by:
• allocating to the processing elements a first subset of the set of computing resources based, at least in part, on a minimum resource requirement for the processing elements, and
• allocating to the processing elements a second subset of the set of computing resources based, at least in part, on an allocation determined using the trained machine learning model;
• wherein the allocating of the second subset is based on scoring of historical resource allocation information associated with the trained machine learning model and selecting CPU core adjustments of processing elements that result in a best score of the scoring.