Managing metadata for a distributed processing system with manager agents and worker agents

A method implemented by data processing apparatus, the method comprising:

• receiving, at a hub device that includes one or more data processing apparatus, a computation specification that includes a set of grains that specifies an initial state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, wherein each grain included in the set of grains comprises metadata that specifies a portion of the initial state for the computation;
• storing, by the hub device, a grain hierarchy that represents a state for the computation based on a grain type associated with each grain included in the set of grains, the grain hierarchy comprising subscription grains for one or more subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and having a corresponding attachment point within the grain hierarchy and a depth that limits a depth of a sub-tree covered by the subscription grain within the grain hierarchy, the sub-tree beginning at the attachment point, and each subscription specifying one or more actions to be performed by the hub device;
• and
• performing, by the hub device and during performance of the computation, at least one of the one or more actions specified by at least one of the one or more subscriptions.

A system comprising:

• a hub device comprising one or more data processing apparatus;
• and
• a data store storing instructions that, when executed by the hub device, cause the hub device to perform operations comprising:
• receiving, at the hub device that includes one or more data processing apparatus, a computation specification that includes a set of grains that specifies an initial state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, wherein each grain included in the set of grains comprises metadata that specifies a portion of the initial state for the computation;
• storing, by the hub device, a grain hierarchy that represents a state for the computation based on a grain type associated with each grain included in the set of grains, the grain hierarchy comprising subscription grains for one or more subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and having a corresponding attachment point within the grain hierarchy and a depth that limits a depth of a sub-tree covered by the subscription grain within the grain hierarchy, the sub-tree beginning at the attachment point, and each subscription specifying one or more actions to be performed by the hub device;
• and
• performing, by the hub device and during performance of the computation, at least one of the one or more actions specified by at least one of the one or more subscriptions.

A non-transitory computer readable medium storing instructions that, when executed by a hub device comprising one or more data processing apparatus, cause the hub device to perform operations comprising:

• receiving, at a hub device that includes one or more data processing apparatus, a computation specification that includes a set of grains that specifies an initial state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, wherein each grain included in the set of grains comprises metadata that specifies a portion of the initial state for the computation;
• storing, by the hub device, a grain hierarchy that represents a state for the computation based on a grain type associated with each grain included in the set of grains, the grain hierarchy comprising subscription grains for one or more subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy and having a corresponding attachment point within the grain hierarchy and a depth that limits a depth of a sub-tree covered by the subscription grain within the grain hierarchy, the sub-tree beginning at the attachment point, and each subscription specifying one or more actions to be performed by the hub device;
• and
• performing, by the hub device and during performance of the computation, at least one of the one or more actions specified by at least one of the one or more subscriptions.