Method and system for event consumer management in an aggregated event platform

A non-transitory computer readable storage medium that stores instructions that when executed by one or more processors causes the one or more processors to perform the following operations:

• receiving, by a first aggregate topic event consumer from a plurality of aggregate topic event consumers based on an aggregate topic, a batch of events stored in an event recordation system according to the aggregate topic, wherein an event from the batch of events is associated with a first initial topic that is different from the aggregate topic and the first initial topic is to be used by an instance of an event consumer from a set of event consumers to consume the event;
• determining, by the first aggregate topic event consumer, the first initial topic associated with one or more events from the batch of events;
• updating, by the first aggregate topic event consumer based on the first initial topic, a state of a first event consumer in a states of event consumers from an idle state to a running state, wherein the idle state indicates that no instance of the first event consumer is in execution in a cluster of servers, wherein the states of event consumers includes entries representing the plurality of aggregate topic event consumers and entries representing event consumers associated with initial topics, wherein a state of each aggregate topic event consumer identified in the entries representing the plurality of aggregate topic event consumers is set to the running state and cannot be updated by an event consumer manager whereas a state of each event consumer identified in the entries representing the event consumers associated with initial topics can be set by the event consumer manager to the running state or the idle state;
• and
• causing, by the event consumer manager based on a determination that the state of the first event consumer has changed from the idle state to the running state, execution of a first instance of the first event consumer on a server of the cluster of servers for consuming events from the event recordation system based on the first initial topic.