Techniques for resource utilization in replication pipeline processing

A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• receiving, by a computing system, a request for a cross-region replication of a file system;
• processing data, by a first pipeline stage of the computing system and a second pipeline stage of the computing system, each pipeline stage of the first pipeline stage and the second pipeline stage comprising a thread pool with a plurality of parallel running threads;
• and
• communicating, by the computing system, between the first pipeline stage and the second pipeline stage through an architecture for enqueuing and dequeuing replication-related jobs for execution, each replication-related job for execution being configured to use a substantially equal amount of resources of the file system, the architecture comprising:
• distributing, by the computing system, the replication-related jobs substantially evenly into a temporary buffer;
• distributing, by the computing system, the replication-related jobs substantially evenly from the temporary buffer to the plurality of parallel running threads for execution;
• and
• executing each of the replication-related jobs, by one of the plurality of parallel running threads, using a substantially equal amount of resources of the file system.

A system, comprising:

• one or more processors;
• and
• one or more computer readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to:
• receive, by a computing system, a request for a cross-region replication of a file system;
• process data by a first pipeline stage of the computing system and a second pipeline stage of the computing system, each pipeline stage of the first pipeline stage and the second pipeline stage comprising a thread pool with a plurality of parallel running threads;
• and
• communicate, by the computing system, between the first pipeline stage and the second pipeline stage through an architecture for enqueuing and dequeuing replication-related jobs for execution, each replication-related job for execution being configured to use a substantially equal amount of resources of the file system, the architecture comprising:
• distribute, by the computing system, the replication-related jobs substantially evenly into a temporary buffer;
• distribute, by the computing system, the replication-related jobs substantially evenly from the temporary buffer to the plurality of parallel running threads for execution;
• and
• execute each of the replication-related jobs, by one of the plurality of parallel running threads, using a substantially equal amount of resources of the file system.