Reinforcement learning-based intelligent control of packet transmissions within ad-hoc networks

A method for an ad-hoc network having a plurality of nodes including a source node and a destination node, the method comprising, at each intermediate one of the nodes besides the destination node:

• measuring a current position and a current velocity of the intermediate node;
• maintaining an estimated position and an estimated velocity for each of the nodes in a neighboring subset of the nodes;
• broadcasting at least one transmitted packet including the current position and the current velocity of the intermediate node;
• correspondingly receiving at least one received packet from at least a proximate one of the nodes, the received packet including a current position and a current velocity of the proximate node;
• updating the estimated position and the estimated velocity of the proximate node with the current position and the current velocity from the received packet;
• calculating a hop duration of a respective reliable link between the intermediate node and each of at least a neighbor one of the nodes in the neighboring subset, including calculating the hop duration from the current position and the current velocity measured for the intermediate node and from the estimated position and the estimated velocity maintained for the neighbor node;
• receiving a payload packet for transmission from the source node to the destination node via the ad-hoc network;
• and
• adaptively deferring a transmission of the payload packet from the intermediate node to the neighbor node of the intermediate node in the ad-hoc network, based on an estimated V-value of the neighbor node, a change in a self V-value of the intermediate node, and the hop duration of the respective reliable link between the intermediate node and the neighbor node, wherein the estimated V-value and the self V-value are determined from a reinforcement learning routing algorithm, the self V-value converges to a current cost of transmitting the payload packet from the intermediate node to the destination node, and the estimated V-value converges to a current cost of transmitting the payload packet from the neighbor node to the destination node via the ad-hoc network,
• wherein the broadcasting includes broadcasting the transmitted packet having a header including a path duration of a network path and a confidence of the network path from the intermediate node to the destination node via the ad-hoc network, wherein the path duration includes a start time and an end time inside the hop duration of the respective reliable link between the intermediate node and the neighbor node, which is a next one of the nodes along the network path, and
• the correspondingly receiving includes from the proximate node that is the neighbor node correspondingly receiving the received packet having the header including a path duration of a portion the network path following the neighbor node and a confidence of the portion from the neighbor node to the destination node via the ad-hoc network, wherein the path duration received includes a start time and an end time.

A system comprising:

• a source node in an ad-hoc network of a plurality of network nodes, which include a destination node, the source node including:
• at least one hardware device processor adapted to:
• measure a current position and a current velocity of the source node;
• maintain an estimated position and an estimated velocity for each of the network nodes in a neighboring subset of the network nodes;
• broadcast at least one transmitted packet including the current position and the current velocity of the source node;
• correspondingly receive at least one received packet from at least a proximate one of the network nodes, the received packet including a current position and a current velocity of the proximate node;
• update the estimated position and the estimated velocity of the proximate node with the current position and the current velocity from the received packet;
• calculate a hop duration of a respective reliable link between the source node and each of at least a neighbor one of the network nodes in the neighboring subset, including calculating the hop duration from the current position and the current velocity measured for the source node and from the estimated position and the estimated velocity maintained for the neighbor node,
• wherein the broadcast includes broadcast the transmitted packet having a header including a path duration of a network path and a confidence of the network path from the source node to the destination node via the ad-hoc network, wherein the path duration includes a start time and an end time inside the hop duration of the respective reliable link between the source node and the neighbor node, which is a next one of the network nodes along the network path, and
• the correspondingly receive includes from the proximate node that is the neighbor node correspondingly receive the received packet having the header including a path duration of a portion the network path following the neighbor node and a confidence of the portion from the neighbor node to the destination node via the ad-hoc network, wherein the path duration received includes a start time and an end time;
• an adaptive transmission controller that controls transmission of a plurality of generated packets by:
• adaptively controlling a generation rate, at the source node, of the plurality of generated packets for transmission to the destination node in the ad-hoc network,
• wherein the generation rate is based on an estimated V-value of the neighbor node, a change in a self V-value of the source node, and the hop duration of the respective reliable link between the source node and the neighbor node, wherein the estimated V-value is determined from a reinforcement learning routing algorithm, the estimated V-value converging to a current energy cost to transmit each of the plurality of generated packets from the neighbor node to the destination node via the ad-hoc network;
• and
• an adaptive defer controller that controls deferral of transmission of a plurality of received packets by:
• adaptively deferring transmission of the plurality of received packets from the source node to the neighbor node of the source node in the ad-hoc network, based on the estimated V-value of the neighbor node, the change in the self V-value of the source node, and the hop duration of the respective reliable link between the source node and the neighbor node, wherein the self V-value is determined from the reinforcement learning routing algorithm, and the self V-value converges to a current energy cost of transmitting each of the plurality of received packets from the source node to the destination node via the ad-hoc network.