# US 12,223,430 B1 · Empirical game theoretic system and method for adversarial decision analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,430 B1
- Document ID: US12223430B1
- Status: Granted
- Kind: B1
- Application number: 17/370,825
- Filed: 2021-07-08
- Granted: 2025-02-11
- Earliest claimed: 2019-03-05
- Assignees: HRL Laboratories, LLC
- Applicants: HRL Laboratories, LLC
- Inventors: Sean Soleyman; Deepak Khosla; Fan H. Hung; Samuel D. Johnson
- Web page: https://patentgrep.com/patent/US12223430B1

## CPC

- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G05B 13/027: {using neural networks only}
- G05B 13/04: involving the use of models or simulators
- G06F 18/2185: {the supervisor being an automated module, e.g. intelligent oracle}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/094: Adversarial learning
- G06N 3/092: Reinforcement learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Described is a system for performing a simulated vehicle control task based on adversarial decision analysis. Empirical game theoretic analyses are performed between an evolving and an adversary population of neural network strategies. Each empirical game theoretic analysis includes using a neuroevolution procedure to perform a fitness-based selection of a strategy in the evolving population that out-performs the adversary population. Using an empirical game theory procedure, the neuroevolution procedure is iteratively run and the selected strategy is added to the adversary population with each iteration, resulting in monotonic strategy improvement with each iteration. Following the empirical game theoretic analyses, a final strategy is selected for the evolving population and the adversary population using a tournament selection procedure. The final strategy is used to train a neural network which is used to perform a simulated vehicle control task.

## Claims

### Claim 1

A system for performing a simulated vehicle control task based on adversarial decision analysis, the system comprising:
one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of:
performing a plurality of empirical game theoretic analyses between an evolving population of neural network strategies and an adversary population of neural network strategies, wherein each empirical game theoretic analysis comprises:
using a neuroevolution procedure, performing a fitness-based selection of a strategy in the evolving population of neural network strategies that out-performs the adversary population of neural network strategies; and
using an empirical game theory procedure, iteratively running the neuroevolution procedure and adding the selected strategy to the adversary population of neural network strategies with each iteration,
wherein addition of the selected strategy to the adversary population of neural network strategies results in monotonic strategy improvement with each iteration;
following the plurality of empirical game theoretic analyses, selecting a final strategy for the evolving population of neural network strategies and the adversary population of neural network strategies using a tournament selection procedure;
using the final strategy, training a neural network; and
using the trained neural network to perform a simulated vehicle control task.

### Claim 2 (depends on 1)

The system as set forth in claim 1, wherein the trained neural network is used to produce a simulated conflict scenario amongst vehicles, wherein the simulated conflict scenario is a graphical representation comprising a course of action for each vehicle of the conflict scenario.

### Claim 3 (depends on 1)

The system as set forth in claim 1, wherein the one or more processors further perform an operation of controlling maneuvering of one or more autonomous vehicles according to the simulated vehicle control task.

### Claim 4 (depends on 2)

The system as set forth in claim 2, wherein the one or more processors further perform an operation of predicting a trajectory of each of the vehicles using both the final strategy for the evolving population of neural network strategies and the final strategy for the adversary population of neural network strategies.

### Claim 5 (depends on 1)

The system as set forth in claim 1, wherein the tournament selection procedure comprises:
evaluating each selected strategy from each iteration of the empirical game theory procedure against selected strategies from all other iterations;
producing a fitness matrix from the evaluation;
using a Nash solver to find a distribution for a symmetric zero-sum strategic form game defined by the fitness matrix; and
outputting a mixed strategy comprising a set of strategies with a sampling distribution.

### Claim 6 (depends on 1)

The system as set forth in claim 1, wherein the tournament selection procedure comprises:
evaluating each selected strategy against a pre-existing adversary strategy to obtain a set of response quality metric results; and
selecting the final strategy according to the set of response quality metric results.

### Claim 7 (depends on 1)

The system as set forth in claim 1, wherein the one or more processors further perform operations of:
evaluating the selected strategy from each iteration of the empirical game theory procedure against randomly sampled strategies from the adversary population of neural network strategies; and
discarding the selected strategy if it fails to result in improvement of the adversarial population of neural network strategies.

### Claim 8 (depends on 1)

The system as set forth in claim 1, wherein the evolving population of neural network strategies is re-initialized with random neural network weight parameters in each empirical game theoretic analysis.

### Claim 9

A computer implemented method for performing a simulated vehicle control task based on adversarial decision analysis, the method comprising an act of:
causing one or more processors to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of:
performing a plurality of empirical game theoretic analyses between an evolving population of neural network strategies and an adversary population of neural network strategies, wherein each empirical game theoretic analysis comprises:
using a neuroevolution procedure, performing a fitness-based selection of a strategy in the evolving population of neural network strategies that out-performs the adversary population of neural network strategies; and
using an empirical game theory procedure, iteratively running the neuroevolution procedure and adding the selected strategy to the adversary population of neural network strategies with each iteration,
wherein addition of the selected strategy to the adversary population of neural network strategies results in monotonic strategy improvement with each iteration;
following the plurality of empirical game theoretic analyses, selecting a final strategy for the evolving population of neural network strategies and the adversary population of neural network strategies using a tournament selection procedure;
using the final strategy, training a neural network; and
using the trained neural network to perform a simulated vehicle control task.

### Claim 10 (depends on 9)

The method as set forth in claim 9, further comprising an operation of using the trained neural network to produce a simulated conflict scenario amongst vehicles, wherein the simulated conflict scenario is a graphical representation comprising a course of action for each vehicle of the conflict scenario.

### Claim 11 (depends on 9)

The method as set forth in claim 9, further comprising an operation of controlling maneuvering of one or more autonomous vehicles according to the simulated vehicle control task.

### Claim 12 (depends on 10)

The method as set forth in claim 10, wherein the one or more processors further perform an operation of predicting a trajectory of each of the vehicles using both the final strategy for the evolving population of neural network strategies and the final strategy for the adversary population of neural network strategies.

### Claim 13 (depends on 9)

The method as set forth in claim 9, wherein the tournament selection procedure comprises:
evaluating each selected strategy from each iteration of the empirical game theory procedure against selected strategies from all other iterations;
producing a fitness matrix from the evaluation;
using a Nash solver to find a distribution for a symmetric zero-sum strategic form game defined by the fitness matrix; and
outputting a mixed strategy comprising a set of strategies with a sampling distribution.

### Claim 14 (depends on 9)

The method as set forth in claim 9, wherein the tournament selection procedure comprises:
evaluating each selected strategy against a pre-existing adversary strategy to obtain a set of response quality metric results; and
the final strategy according to the set of response quality metric results.

### Claim 15 (depends on 9)

The method as set forth in claim 9, wherein the one or more processors further perform operations of:
evaluating the selected strategy from each iteration of the empirical game theory procedure against randomly sampled strategies from the adversary population of neural network strategies; and
discarding the selected strategy if it fails to result in improvement of the adversarial population of neural network strategies.

### Claim 16 (depends on 9)

The method as set forth in claim 9, wherein the evolving population of neural network strategies is re-initialized with random neural network weight parameters in each empirical game theoretic analysis.

### Claim 17

A computer program product for performing a simulated vehicle control task based on adversarial decision analysis, the computer program product comprising:
computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors for causing the processor to perform operations of:
performing a plurality of empirical game theoretic analyses between an evolving population of neural network strategies and an adversary population of neural network strategies, wherein each empirical game theoretic analysis comprises:
using a neuroevolution procedure, performing a fitness-based selection of a strategy in the evolving population of neural network strategies that out-performs the adversary population of neural network strategies; and
using an empirical game theory procedure, iteratively running the neuroevolution procedure and adding the selected strategy to the adversary population of neural network strategies with each iteration,
wherein addition of the selected strategy to the adversary population of neural network strategies results in monotonic strategy improvement with each iteration;
following the plurality of empirical game theoretic analyses, selecting a final strategy for the evolving population of neural network strategies and the adversary population of neural network strategies using a tournament selection procedure;
using the final strategy, training a neural network; and
using the trained neural network to perform a simulated vehicle control task.

### Claim 18 (depends on 17)

The computer program product as set forth in claim 17, further comprising instructions for causing the one or more processors to further perform an operation of predicting a trajectory of each vehicle in the vehicle simulation task using both the final strategy for the evolving population of neural network strategies and the final strategy for the adversary population of neural network strategies.

### Claim 19 (depends on 17)

The computer program product as set forth in claim 17, wherein the tournament selection procedure comprises:
evaluating each selected strategy from each iteration of the empirical game theory procedure against selected strategies from all other iterations;
producing a fitness matrix from the evaluation;
using a Nash solver to find a distribution for a symmetric zero-sum strategic form game defined by the fitness matrix; and
outputting a mixed strategy comprising a set of strategies with a sampling distribution.

### Claim 20 (depends on 17)

The computer program product as set forth in claim 17, wherein the tournament selection procedure comprises:
evaluating each selected strategy against a pre-existing adversary strategy to obtain a set of response quality metric results; and
selecting the final strategy according to the set of response quality metric results.

### Claim 21 (depends on 17)

The computer program product as set forth in claim 17, further comprising instructions for causing the one or more processors to further performs of:
evaluating the selected strategy from each iteration of the empirical game theory procedure against randomly sampled strategies from the adversary population of neural network strategies; and
discarding the selected strategy if it fails to result in improvement of the adversarial population of neural network strategies.

### Claim 22 (depends on 17)

The computer program product as set forth in claim 17, wherein the evolving population of neural network strategies is re-initialized with random neural network weight parameters in each empirical game theoretic analysis.