Data transfer in a collaborative file sharing system

An apparatus for distributing log information in a cross-enterprise file collaboration system, the system including one or more server computers coupled to one or more client devices, wherein the client devices include a first group of client devices on a first network and a second group of client devices on a second network, the apparatus comprising:

• a digital processor coupled to a display and to a non-transitory processor-readable storage device, wherein the non-transitory processor-readable storage device includes one or more instructions executable by the digital processor to perform the following acts:
• receiving log information from a service that is in communication with a web application in response to an operation of a client device;
• storing log information in a message broker;
• using a replicator of the one or more server computers to fetch at least a portion of the log information from the message broker, resulting in fetched log information in response thereto, wherein fetching the at least a portion of the log information comprises:
• using a replicator worker in communication with one or more fetchers to receive a portion of a message stream from the message broker, the one or more fetchers in communication with one or more committers, the replication worker comprising one of a plurality of replication workers for the replicator of the one or more servers, wherein the replication worker retrieves batches of data from the message broker and performs task assignments received from the replicator,
• using the one or more fetchers to selectively cache date from the message broker and share the cached data among one or more committers, each of the one or more fetchers comprising a child node of the replication worker, wherein each fetcher monitors and handles a predefined partition of a message topic of a plurality of message topics for the log information, and
• using the one or more committers to write at least a portion of the fetched log information to one or more targets, each of the one or more committers comprising a child node of at least one of the one or more fetchers, wherein each committer replicates data from the message broker to one or more targets for the committer and performs, by execution of a custom code plug-in, one or more overhead functions for the one or more targets of the committer, and wherein the log information written to the one or more targets is also replicated across two or more data centers by the one or more committers.

A non-transitory processor-readable storage device including instructions executable by a digital processor, the processor-readable storage device including one or more instructions for:

• receiving log information from a service that is in communication with a web application in response to an operation of a client device;
• storing log information in a message broker;
• using a replicator of the one or more server computers to fetch at least a portion of the log information from the message broker, resulting in fetched log information in response thereto, wherein fetching the at least a portion of the log information comprises:
• using a replicator worker in communication with one or more fetchers to receive a portion of a message stream from the message broker, the one or more fetchers in communication with one or more committers, the replication worker comprising one of a plurality of replication workers for the replicator of the one or more servers, wherein the replication worker retrieves batches of data from the message broker and performs task assignments received from the replicator,
• using the one or more fetchers to selectively cache date from the message broker and share the cached data among one or more committers, each of the one or more fetchers comprising a child node of the replication worker, wherein each fetcher monitors and handles a predefined partition of a message topic of a plurality of message topics for the log information, and
• using the one or more committers to write at least a portion of the fetched log information to one or more targets, each of the one or more committers comprising a child node of at least one of the one or more fetchers, wherein each committer replicates data from the message broker to one or more targets for the committer and performs, by execution of a custom code plug-in, one or more overhead functions for the one or more targets of the committer, and wherein the log information written to the one or more targets is also replicated across two or more data centers by the one or more committers.