# US 2024/0205095 A1 · Distributed Deep Reinforcement Learning Framework for Software-Defined Unmanned Aerial Vehicle Network Control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0205095 A1
- Document ID: US20240205095A1
- Status: Published application
- Kind: A1
- Application number: 17/846,197
- Filed: 2022-06-22
- Published: 2024-06-20
- Earliest claimed: 2021-06-28
- Assignees: Northeastern University
- Applicants: Northeastern University
- Inventors: Lorenzo BERTIZZOLO; Tommaso MELODIA; Salvatore D'ORO; Hai CHENG
- Web page: https://patentgrep.com/patent/US20240205095A1

## CPC

- H04L 41/122: of virtualised topologies, e.g. software-defined networks [SDN] or network function virtualisation [NFV]
- H04W 84/06: Airborne or Satellite Networks (space-based or airborne stations H04B7/185)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

Provided herein are systems for controlling a network of distributed non-terrestrial nodes including a control framework operative to train and control a plurality of the non-terrestrial nodes, the control framework including a control interface in communication with a network operator to receive one or more specified control objectives, and a learning engine operative to train a virtual non-terrestrial network, wherein the control framework is further operative to transfer knowledge gained through the training of the virtual non-terrestrial network to the network of distributed non-terrestrial nodes as data-driven logic unit configurations tailored for the specified control objectives.

## Claims

### Claim 1

A system for controlling a network of distributed non-terrestrial nodes comprising:
a control framework, comprising one or more processors and memory, operative to train and control a plurality of the non-terrestrial nodes, the control framework comprising:
a control interface in communication with a network operator to receive one or more specified control objectives, and
a learning engine operative to train a virtual non-terrestrial network;
wherein the control framework is further operative to transfer knowledge gained through the training of the virtual non-terrestrial network to the network of distributed non-terrestrial nodes as data-driven logic unit configurations tailored for the specified control objectives.

### Claim 2 (depends on 1)

The system of claim 1, wherein the non-terrestrial nodes are operative to execute the received data-driven logic unit configurations on-board based on local state information.

### Claim 3 (depends on 1)

The system of claim 1, wherein the non-terrestrial nodes are operative to communicate with each other to exchange local information for facilitating the execution of the received data-driven logic unit configurations.

### Claim 4 (depends on 1)

The system of claim 1, wherein the virtual non-terrestrial network includes a number of virtual non-terrestrial nodes each corresponding to one or more of the non-terrestrial nodes.

### Claim 5 (depends on 4)

The system of claim 4, wherein each of the virtual non-terrestrial nodes, when trained, corresponds to one or more of the data-driven logic unit configurations.

### Claim 6 (depends on 1)

The system of claim 1, wherein each of the data-driven logic unit configurations corresponds to one or more of the non-terrestrial nodes.

### Claim 7 (depends on 6)

The system of claim 6, wherein each of a plurality of the non-terrestrial nodes comprises an unmanned aerial vehicle (UAV) including:
a radio front-end operative for electronic communication with at least one other of the non-terrestrial nodes;
a motion front-end operative to control a motion of the UAV; and
a drone programmable protocol stack (DPPS) having a data-driven logic unit for operating the UAV.

### Claim 8 (depends on 7)

The system of claim 7, wherein the data-driven logic unit of the DPPS includes a neural network and the data-driven logic unit is a neural network configuration.

### Claim 9 (depends on 8)

The system of claim 8, wherein the neural network is at least one of a deep learning neural network, a reinforcement learning neural network, or a deep reinforcement learning neural network.

### Claim 10 (depends on 9)

The system of claim 9, wherein the neural network is a deep reinforcement learning neural network and the DPPS is a drone programmable protocol Stack (DRL DPPS).

### Claim 11 (depends on 10)

The system of claim 10, wherein the DRL DPPS of each UAV includes a DRL plane operative to execute a deep reinforcement learning neural network according to the corresponding neural network configuration using inputs corresponding to one or more current motion and network conditions of at least a portion of the network of distributed non-terrestrial nodes to determine motion and networking policies.

### Claim 12 (depends on 11)

The system of claim 11, wherein the DRL DPPS of each UAV includes a data plane operative to:
reconfigure motion control and networking stack parameters of the UAV responsive to the determined motion and networking policies; and
determine the current motion and network conditions of the at least a portion of the network of distributed non-terrestrial nodes.

### Claim 13 (depends on 12)

The system of claim 12, wherein each UAV includes a register plane operative to:
store the motion and networking policies determined by the DRL plane in one or more policy lookup tables; and
store the current motion and network conditions determined by the data plane in one or more network state lookup tables.

### Claim 14 (depends on 13)

The system of claim 13, wherein the DRL plane is further operative to retrieve the current motion and network conditions from the one or more network state lookup tables of the register plane.

### Claim 15 (depends on 13)

The system of claim 13, wherein the data plane is operative to retrieve the motion and networking policies from the one or more policy lookup tables.

### Claim 16 (depends on 13)

The system of claim 13, wherein the data plane includes a plurality of operative layers, each of the operative layers corresponding to a networking or motion functionality.

### Claim 17 (depends on 16)

The system of claim 16, wherein the plurality of operative layers includes at least two of a motion layer, a physical layer, a MAC layer, a network layer, a transport layer, and an application layer.

### Claim 18 (depends on 17)

The system of claim 17, further comprising a driver plane including:
a flight control firmware operative to receive motion control information from the motion layer of the data plane and to operate the motion front-end in accordance with the motion control information; and
a software-defined radio operative to receive physical control information from the physical layer of the data plane and to operate the radio front-end in accordance with the physical control information.

### Claim 19 (depends on 16)

The system of claim 16, wherein the register plane includes at least one network state lookup table corresponding to each of the operative layers.

### Claim 20 (depends on 16)

The system of claim 16, wherein the register plane includes at least one policy lookup table corresponding to each of the operative layers.

### Claim 21 (depends on 10)

The system of claim 10, wherein the learning engine further comprises:
a virtual training environment operative to generate and train a corresponding one of the virtual non-terrestrial nodes for each of the UAVs, each of the corresponding virtual non-terrestrial nodes including a virtual UAV having a DRL DPPS;
a virtual testing environment operative to test the trained virtual UAVs; and
a virtual emulation environment operative to emulate at least a motion layer and a physical layer of the DRL DPPS within the virtual training environment and the virtual testing environment.

### Claim 22 (depends on 21)

The system of claim 21, wherein the virtual emulation environment is in communication with a MAC layer of a data plane of the DRL DPPS of the virtual non-terrestrial node, one or more policy lookup tables of a register plane of the DRL DPPS of the virtual non-terrestrial node, and one or more network state lookup tables of the register plane of the DRL DPPS of the virtual non-terrestrial node.

### Claim 23 (depends on 21)

The system of claim 21, wherein the virtual emulation environment includes a common open research emulator (CORE) integrated with an extendable mobile ad-hoc network emulator (EMANE).
