Processing large requests in data storage systems with limited/constant buffer sizes

A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

• allocating an available receive slot of an incoming network queue to a network packet of a request received at a computing node, the computing node forming part of a distributed computing system implementing a distributed data storage application, wherein the distributed computing system implements a consensus protocol to ensure that replicas of a state machine of the distributed data storage application are kept in a consistent state based on a consensus among computing nodes forming the distributed computing system;
• processing receipt of the network packet using the allocated receive slot;
• allocating an available free I/O control block of a free I/O control block queue and an available request slot from a free request queue to the network packet after completion of the receipt processing, the free I/O control block queue and free request queue respectively having a same number of I/O queue blocks and request slots as the incoming network queue has receive slots;
• completing an I/O task required for the network packet using the allocated I/O control block;
• processing the network packet using the free request queue;
• assigning a set of processing slots within an additional processing queue for processing of transmission of network packets between the computing nodes of the distributed computing system and for replication of the finite state machine to implement the consensus protocol, the number of slots in the set of slots equal to a number of slots of the incoming network queue, the additional processing queue separate from the incoming network queue;
• and
• allocating an available send slot of an outgoing network queue to send a reply to a client machine from which the request was received.

A system comprising computer hardware configured to perform operations comprising:

• allocating an available receive slot of an incoming network queue to a network packet of a request received at a computing node, the computing node forming part of a distributed computing system implementing a distributed data storage application, wherein the distributed computing system implements a consensus protocol to ensure that replicas of a state machine of the distributed data storage application are kept in a consistent state based on a consensus among computing nodes forming the distributed computing system;
• processing receipt of the network packet using the allocated receive slot;
• allocating an available free I/O control block of a free I/O control block queue and an available request slot from a free request queue to the network packet after completion of the receipt processing, the free I/O control block queue and free request queue respectively having a same number of I/O queue blocks and request slots as the incoming network queue has receive slots;
• completing an I/O task required for the network packet using the allocated I/O control block;
• processing the network packet using the free request queue;
• assigning a set of processing slots within an additional processing queue for processing of transmission of network packets between the computing nodes of the distributed computing system and for replication of the finite state machine to implement the consensus protocol, the number of slots in the set of slots equal to a number of slots of the incoming network queue, the additional processing queue separate from the incoming network queue;
• and
• allocating an available send slot of an outgoing network queue to send a reply to a client machine from which the request was received.