Federated deep reinforcement learning-assisted UAV trajectory planning against hostile defense system

A method for real-time path planning for unmanned aerial vehicles (UAVs), the method comprising:

• a) receiving experience data of a plurality of UAVs in an environment;
• b) sorting the experience data into critical experience data and non-critical experience data;
• c) storing the critical experience data in a first reply memory buffer that is a significant reply memory buffer (SRMB) and storing the non-critical experience data in a second reply memory buffer;
• d) performing a sample technique on the critical experience data from the first reply memory buffer and the non-critical experience data from the second reply memory buffer to obtain sample data;
• e) providing the sample data to a Q-network, the Q-network being a first deep reinforcement learning (DRL) neural network;
• f) copying weights from the Q-network to a target network after a predetermined number of episode steps of the Q-network, the target network being a second DRL neural network;
• g) using the target network to predict future Q-values for future states;
• h) calculating a loss of a prediction of the Q-network based on the future Q-values;
• i) calculating an updated path model for the plurality of UAVs based on the prediction of the Q-network and the loss of the prediction of the Q-network;
• j) generating, by the plurality of UAVs, updated experience data using the updated path model;
• k) repeating steps a)-j), starting with the updated experience data, until the updated path model meets a predetermined threshold to obtain a final path model, and
• l) maneuvering the plurality of UAVs based on the final path model,
• the performing of the sample technique comprising setting a value for a percentage of critical experience data from the first reply memory buffer that makes up the sample data, where a remainder of the sample data is the non-critical experience data from the second reply memory buffer.

A system for real-time path planning for unmanned aerial vehicles (UAVs), the system comprising:

• a processor;
• a display in operable communication with the processor;
• and
• a machine-readable medium in operable communication with the processor and the display and having instructions stored thereon that, when executed by the processor, perform the following steps:
• a) receiving experience data of a plurality of UAVs in an environment;
• b) sorting the experience data into critical experience data and non-critical experience data;
• c) storing the critical experience data in a first reply memory buffer that is a significant reply memory buffer (SRMB) and storing the non-critical experience data in a second reply memory buffer;
• d) performing a sample technique on the critical experience data from the first reply memory buffer and the non-critical experience data from the second reply memory buffer to obtain sample data;
• e) providing the sample data to a Q-network, the Q-network being a first deep reinforcement learning (DRL) neural network;
• f) copying weights from the Q-network to a target network after a predetermined number of episode steps of the Q-network, the target network being a second DRL neural network;
• g) using the target network to predict future Q-values for future states;
• h) calculating a loss of a prediction of the Q-network based on the future Q-values;
• i) calculating an updated path model for the plurality of UAVs based on the prediction of the Q-network and the loss of the prediction of the Q-network;
• j) generating, by the plurality of UAVs, updated experience data using the updated path model;
• k) repeating steps a)-j), starting with the updated experience data, until the updated path model meets a predetermined threshold to obtain a final path model;
• and
• l) maneuvering the plurality of UAVs based on the final path model;
• and
• m) displaying the final path model on the display,
• the environment being a simulated environment,
• the experience data being generated using federated learning (FL) of DRL agents,
• the environment comprising a hostile defense system comprising a static defense system and a dynamic defense system,
• the sorting of the experience data comprising classifying experiences where a UAV of the plurality of UAVs ended on a goal or an obstacle as critical experience data and classifying experiences where a UAV of the plurality of UAVs did not end on a goal or an obstacle as non-critical experience data,
• the performing of the sample technique comprising setting a value for a percentage of critical experience data from the first reply memory buffer that makes up the sample data, where a remainder of the sample data is the non-critical experience data from the second reply memory buffer,
• the generating of updated experience data comprising setting an exploitation parameter in a range of from 0 to 1, where a higher value of the exploitation parameter indicates a higher probability that the plurality of UAVs will take random actions during the generating of updated experience data, and where a lower value of the exploitation parameter indicates a higher probability that the plurality of UAVs will take actions by consulting the Q-network during the generating of updated experience data, and
• the plurality of UAVs comprising at least 10 UAVs.