System and method for data transfer and request handling among a plurality of resources

A computerized system of main controller that each node uses to process packets comprising:

• wherein each software process within a group of independently operating processes in a node may contain an embedded main controller, wherein the main controller enables data transmission across one or more interconnected network segments while adhering to a predetermined protocol, wherein the main controller is configured to process packets, wherein the main controller comprises a plurality of receiving threads while receiving or transmitting a plurality of packets, and wherein when packets are received, a payload of the packet is placed per flow into a payload buffer, which notifies a suitable context queue by identifying a connection and a number of bytes received, and wherein the main controller further comprises:
• a policy controller configured to handle how and when a request for data and computation is performed, relayed, and cached, based on a localized cost estimation procedure, and
• a mode selector configured to calibrate a sending rate, a transmission window size, and a payload buffer size, based on a given choice complexity parameter that optimizes for a high throughput, a loss avoidance and a low latency,
• wherein the policy controller and mode selector execute control decisions synchronously with packet reception events within the main controller, based on per-flow receive-side payload buffer state, prior to any transmit-side scheduling or forwarding,
• wherein transmit scheduling parameters are derived from a receive-side context queue associated with a flow and are applied only after receive-side processing completes, such that transmit context state is updated lazily and independently of receive-side lock acquisition, and
• wherein all control decisions are performed using only node-local and thread-local state, without requiring global network topology information, centralized policy servers, or coordination across multiple nodes.