Parallel prefetching log/meta stream sub-portions to recreate partition states in a distributed computing system

A computer-implemented method, comprising:

• receiving an instruction to generate, at a second node, a new instance of a partition in an operational state that is duplicative of a previous instance of the partition at a first node, the operational state resulting from a plurality of transactions occurring subsequent to a previous operational state of the partition;
• analyzing metadata, corresponding to the previous operational state, to identify a checkpoint of a log stream of transactions that have occurred with respect to the previous instance of the partition at the first node;
• determining, based at least in part on the checkpoint, a plurality of sequence identifiers (IDs) associated with a portion of the log stream that corresponds to the plurality of transactions, wherein individual sequence IDs of the plurality of sequence IDs correspond to individual transactions of the plurality of transactions;
• parsing the portion of the log stream into a plurality of sub-portions based at least partially on the plurality of sequence IDs;
• causing individual prefetch buffers, of a plurality of prefetch buffers, to simultaneously load individual sub-portions of the plurality of sub-portions;
• causing the new instance of the partition to enter the previous operational state at the second node;
• and
• transforming, at the second node, the new instance of the partition into the operational state by replaying, to the new instance of the partition in the previous operational state, the plurality of transactions by sequentially processing the individual sub-portions according to the plurality of sequence IDs.

A system, comprising:

• one or more processors;
• and
• a memory in communication with the one or more processors, the memory having computer-readable instructions stored thereupon that, when executed by the one or more processors, cause the one or more processors to:
• receive an instruction to generate a new instance of a partition in an operational state that is duplicative of a previous instance of the partition;
• analyze metadata to identify a plurality of transactions that have occurred with respect to a previous operational state of the previous instance to result in the previous instance entering the operational state, wherein individual transactions of the plurality of transactions correspond to individual sequence identifiers (IDs);
• parse the plurality of transactions into a plurality of subsets of transactions that includes at least:
• a first subset of transactions, and a second subset of transactions that is subsequent to the first subset of transactions;
• cause individual prefetch buffers, of a plurality of prefetch buffers, to simultaneously load individual subsets of transactions of the plurality of subsets of transactions;
• based at least in part on a determination that a first prefetch buffer has completely loaded the first subset of transactions, commence replay of the first subset of transactions to the new instance while the new instance is in the previous operational state and prior to a second prefetch buffer fully loading the second subset of transactions.