# US 2021/0067417 A1 · Methods, systems and appratuses for optimizing the bin selection of a network scheduling and configuration tool (nst) by bin allocation, demand prediction and machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0067417 A1
- Document ID: US20210067417A1
- Status: Published application
- Kind: A1
- Application number: 16/554,280
- Filed: 2019-08-28
- Published: 2021-03-04
- Earliest claimed: 2019-08-28
- Assignees: Honeywell International Inc.
- Applicants: Honeywell International Inc.
- Inventors: Benjamin F. DeLay; Srivatsan Varadarajan
- Web page: https://patentgrep.com/patent/US20210067417A1

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/092: Reinforcement learning
- G06N 5/043: Distributed expert systems; Blackboards
- H04L 41/0895: Configuration of virtualised networks or elements, e.g. virtualised network function or OpenFlow elements
- H04L 41/16: using machine learning or artificial intelligence
- H04L 49/25: Routing or path finding in a switch fabric
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Methods, systems and apparatuses to enable an optimum bin selection by implementing a neural network with a network scheduling and configuration tool (NST), the method includes: configuring an agent with a critic function from neural networks wherein the agent neural network represents each bin of the collection of bins in the network that performs an action, and a critic function evaluates a criteria of success for performing the action; processing, by a scheduling algorithm, the VLs by the NST; determining one or more reward functions using global quality measurements based on criteria comprising: a lack of available bins, a lack of available VLs, and successfully scheduling operations of a VL into a bin; and training the network based on a normalized state model of the scheduled network by using input data sets to arrive at an optimum bin selection.

## Claims

### Claim 1

A method to enable an optimum bin selection by implementing a neural network with a network scheduling and configuration tool (NST), the method comprising:
receiving, as input, a complete set of virtual links (VLs) by the NST of VLs scheduled across nodes in a network;
establishing a collection of bins that corresponds to the smallest harmonic period allowing full network traversal of a time-triggered traffic packet in the network for determining an initial bin collection size for sending the VL data by the NST;
configuring an agent with a critic function from neural networks wherein each agent neural network represents each bin of the collection of bins in the network that performs an action, and a critic function evaluates a criteria of success for performing the action;
processing, by a scheduling algorithm, the VLs by the NST wherein the scheduling algorithm normalizes a state of a scheduled network into a set of vectors of bins of the collection of bins with quality aspects defined by state variables for inclusion in the scheduling algorithm;
determining one or more reward functions using global quality measurements based on criteria comprising: a lack of available bins, a lack of available VLs, and successfully scheduling operations of a VL into a bin; and
training the network based on a normalized state model of the scheduled network by using input data sets at predetermined points in a scheduling timeline for configuring sets of vectors of bins with learnt quality aspects to arrive at an optimum bin selection.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
performing subsequent bin selection by using a trained network represented by the neural network and by scheduling using the scheduling algorithm of each set of input VLs.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
evaluating by the critic function, the criteria of success based on a set of factors comprising: a lack of available bins, a lack of available VLs, and a completion of the scheduling of a VL into a particular bin.

### Claim 4 (depends on 1)

The method of claim 1 wherein the optimum bin selection using a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) application to adjust a time slot selection algorithm policy for each bin.

### Claim 5 (depends on 4)

The method of claim 4 further comprising:
employing, by the MADDPG application, two neural networks per agent comprising an actor and an actor's target model, and also employing two global neural networks comprising a centralized critic and a centralized critic's target model to determine optimum bin selection.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
treating, by the MADDPG application, each bin as an independent agent, and performing calculations of demand values at the beginning of a scheduling cycle over a duplicate set of global bins which represents an entire schedule cycle using a bin demand property on each bin of a global bin set.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
configuring an action space for the actor which comprises available bins that fall within a current VL's green zone wherein the current VL's green zone is a current time box within a period of scheduling a VL activity based on available latency-related constraints.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
calculating the reward functions globally by the centralized critic for each actor and configuring using various quality vectors based on a goal of the scheduling algorithm.

### Claim 9 (depends on 8)

The method of claim 8, wherein the goal is constrained by a set of factors including the lack of available bins, the packing efficiency of the bins, and the remaining time in the bins after scheduling.

### Claim 10

A system to enable an optimum bin selection by implementing neural networks with a network scheduling and configuration tool (NST), the system comprising:
the NST configured to:
receive, as input, a complete set of virtual links (VLs) by the NST of VLs scheduled across nodes in a network;
establish a collection of bins which corresponds to the smallest harmonic period allowing full network traversal of a time-triggered traffic packet in the network to determine an initial bin collection size for sending the VL data by the NST;
configure an agent with a critic function from neural networks wherein each agent neural network represents each bin of the collection of bins in the network that performs an action, and the critic function evaluates a criteria of success to perform the action;
process, by executing a schedule algorithm, the VLs by the NST wherein the schedule algorithm normalizes a state of a scheduled network into a set of vectors of bins of the collection with quality aspects defined by state variables for inclusion in the scheduling algorithm;
determine one or more reward functions using global quality measurements based on criteria comprising: a lack of available bins, a lack of available VLs, and successfully scheduling operations of a VL into a bin; and
train the network based on a normalized state model of the scheduled network by using input data sets at predetermined points in a scheduling timeline to configure sets of vectors of bins with learnt quality aspects to arrive at an optimum bin selection.

### Claim 11 (depends on 10)

The system of claim 10, further comprising:
the NST configured to:
perform subsequent bin selections by using a trained network of the neural network and to schedule using the schedule algorithm with quality aspects for each set of input VLs.

### Claim 12 (depends on 11)

The system of claim 11, further comprising:
the NST configured to:
evaluate by the critic function, the criteria of success based on a set of factors comprising: a lack of available bins, a lack of available VLs, and a completion of the scheduling of a VL into a particular bin.

### Claim 13 (depends on 12)

The system of claim 12, wherein the optimum bin selection implements a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) application to adjust a time slot selection algorithm policy for each bin.

### Claim 14 (depends on 13)

The system of claim 13, further comprising:
the NST configured to:
employ, via the MADDPG application, two neural networks per agent comprising an actor and an actor's target model, and also employ two global neural networks comprising a centralized critic and a centralized critic's target model to determine the optimum bin selection.

### Claim 15 (depends on 14)

The system of claim 14, further comprising:
the NST configured to:
treat, via the MADDPG application, each bin as an independent agent, and perform calculations of demand values at a beginning of a scheduling cycle over a duplicate set of global bins which represent an entire schedule cycle using a bin demand property on each bin of a global bin set.

### Claim 16 (depends on 15)

The system of claim 15, further comprising:
an action space for the actor comprises available bins that fall within a current VL's green zone wherein the current VL's green zone is a current time box within a period to schedule a VL activity based on available latency-related constraints.

### Claim 17 (depends on 16)

The system of claim 16, further comprising:
the NST configured to:
calculate the reward functions globally by the centralized critic for each actor and to use quality vectors which correspond to a goal of the schedule algorithm.

### Claim 18 (depends on 17)

The system of claim 17, wherein the goal is constrained by a set of factors including the lack of available bins, the packing efficiency of the bins of the collection, and the remaining time in the bins after scheduling.

### Claim 19

An apparatus to enable an optimum bin selection by implementing neural networks with a network scheduling and configuration tool (NST), the apparatus comprises:
the NST configured to:
receive, as input, a complete set of virtual links (VLs) by the NST of VLs scheduled across nodes in a network;
establish a collection of bins which corresponds to the smallest harmonic period allowing full network traversal of a time-triggered traffic packet in the network to determine an initial bin collection size for sending the VL data by the NST;
configure an agent with a critic function from neural networks wherein the agent neural network represents each bin of the collection in the network that performs an action, and the critic function evaluates a criteria of success to perform the action;
employ, via a MADDPG application, two neural networks per agent comprising an actor and an actor's target model, and also employ two global neural networks comprising a centralized critic and a centralized critic's target model to determine the optimum bin selection;
process, by executing a schedule algorithm, the VLs by the NST wherein the schedule algorithm normalizes a state of a scheduled network into a set of vectors of bins of the collection with quality aspects defined by state variables for inclusion in the scheduling algorithm;
determine one or more reward functions using global quality measurements based on criteria comprising: a lack of available bins, a lack of available VLs, and successfully scheduling operations of a VL into a bin; and
train the network based on a normalized state model of the scheduled network by using input data sets at predetermined points in a scheduling timeline to configure sets of vectors of bins with learnt quality aspects to arrive at an optimum bin selection.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) application adjusts a time slot selection algorithm policy for each bin for optimum bin selection, and treats, via the MADDPG application, each bin as an independent agent to perform calculations of demand values at a beginning of a scheduling cycle over a duplicate set of global bins which represent an entire schedule cycle using a bin demand property on each bin of a global bin set.
