Formula-based load evaluation in distributed streaming platform for real-time applications

One or more computer-readable media that are non-transitory and that store one or more programs, wherein the one or more programs, when executed, instructs one or more processors to perform the following operations:

• receive an application that runs on a streaming platform, wherein the application is structured as a directed acyclic graph (DAG) with instances of operators as nodes and streams as edges between nodes and wherein the application is associated with a pre-defined hint that is a key-value pair;
• launch the application by assigning the instances of operators to containers provided by streaming platform and initiate the streams;
• read a value for the pre-defined hint and transmitting the value to the application through an application programming interface (API) exposed by the streaming platform;
• receive a request, based at least in part on the value, from the application through the API to make a dynamic adjustment to one or more of the instances;
• and
• make the dynamic adjustment and re-launching the application using a recovery policy, wherein the dynamic adjustment includes a dynamic partition of at least one instance into a plurality of instances, wherein the at least one instance reads from and/or writes to a hash table or hash map associated with the instance, and wherein the pre-defined hint is a count of entries in the hash table or hash map.

A method, comprising the operations of:

• receiving an application that runs on a streaming platform, wherein the application is structured as a directed acyclic graph (DAG) with instances of operators as nodes and streams as edges between nodes and wherein the application is associated with a pre-defined hint;
• launching the application by assigning the instances of operators to containers provided by streaming platform and initiating the streams;
• reading a value for the pre-defined hint and transmitting the value to the application through an application programming interface (API) exposed by the streaming platform;
• receiving a request, based at least in part on the value, from the application through the API to make a dynamic adjustment to one or more of the instances;
• and
• making the dynamic adjustment and re-launching the application, wherein the dynamic adjustment includes a dynamic partition of at least one instance into a plurality of instances, wherein the at least one instance reads from and/or writes to a hash table or hash map associated with the instance, wherein the pre-defined hint is a count of entries in the hash table or hash map, and wherein each of the operations is executed by one or more processors in real time or near real time rather than offline.