Data Object Delivery for Distributed Cluster Computing

A cluster computing system comprising:

• a worker device in communication with a master device;
• wherein the worker device is configured to:
• subscribe to a dataset;
• receive, via a first process executing on the worker device and from the master device, the dataset;
• store the dataset on a local storage media of the worker device;
• wherein the storing the dataset comprises:
• determining, based on metadata associated with the dataset, that a quantity of updates to the dataset with a time period satisfies a threshold;
• and
• based on the determining that the quantity of updates satisfies the threshold, storing the dataset in the local storage media;
• execute, via one or more processes different from the first process, a plurality of applications to perform a task on the dataset, wherein each application of the plurality of applications is configured to share, with each other application of the plurality of applications, the dataset from the local storage media;
• receive, based on the subscribing and from the master device, a data update message indicating a change to the dataset;
• and
• update, based on the data update message, the dataset on the local storage media.

A non-transitory machine-readable medium storing instructions, that when executed by one or more processors, cause the one or more processors to:

• subscribe, by a worker device, to a dataset;
• receive, via a first process executing on the worker device and from a master device, the dataset;
• store the dataset on a local storage media of the worker device;
• wherein the storing the dataset comprises;
• determining based on metadata associated with the dataset, that a quantity of updates to the dataset within a time period satisfies a threshold;
• and
• based on the determining that the quantity of updates satisfies the threshold, store the dataset on a local storage media;
• execute, via one or more processes different from the first process, a plurality of applications to perform a task on the dataset, wherein each application of the plurality of applications is configured to share, with each other application of the plurality of applications, the dataset from the local storage media;
• receive, based on the subscribing and from the master device, a data update message indicating a change to the dataset;
• and
• update, based on the data update message, the dataset on the local storage media.