System and method for maintaining tenant data isolation in a messaging service

A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process for maintaining tenant isolation in a messaging service, the process comprising:

• receiving, in at least one source topic, records sent by a plurality of producer systems associated with respective ones of a plurality of tenants, wherein each of the plurality of tenants is associated with a unique tenant identifier (ID) and wherein each of the at least one source topic includes a plurality of partitions each of which is adapted to store records received from at least one of the tenants in the order of receipt and each respective record contains the unique tenant ID of the tenant sourcing the record;
• partitioning the received records into a plurality of partitions in an intermediate topic based on the respective tenant IDs of respective tenants that sourced the records;
• grouping, for each of the plurality of partitions in the intermediate topic, records within the partition into an isolated batch, wherein the records in each isolated batch belong to the same tenant;
• and
• placing the isolated batches in a destination topic to be consumed by a consumer system, wherein the isolated batches are placed in the destination topic in a round-robin manner;
• wherein each of the source, intermediate, and destination topics are realized as respective data structures stored in memory and wherein each partition of any topic is a memory area within that topic.

A system for maintaining tenant isolation in a messaging service, comprising:

• a processing circuitry;
• and
• a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:
• receive, in at least one source topic, records sent by a plurality of producer systems associated with respective ones of a plurality of tenants, wherein each of the plurality of tenants is associated with a unique tenant identifier (ID) and wherein each of the at least one source topic includes a plurality of partitions each of which is adapted to store records received from at least one of the tenants in the order of receipt and each respective record contains the unique tenant ID of the tenant sourcing the record;
• partition the received records into a plurality of partitions in an intermediate topic based on the respective tenant IDs of respective tenants that sourced the records;
• group, for each of the plurality of partitions in the intermediate topic, records within the partition into an isolated batch, wherein the records in each isolated batch belong to the same tenant;
• and
• place the isolated batches in a destination topic to be consumed by a consumer system, wherein the isolated batches are placed in the destination topic in a robin-round manner;
• wherein each of the source, intermediate, and destination topics are realized as respective data structures stored in the memory and wherein each partition of any topic is a memory area within that topic.