# US 2021/0334671 A1 · Learning Agent

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334671 A1
- Document ID: US20210334671A1
- Status: Published application
- Kind: A1
- Application number: 17/243,445
- Filed: 2021-04-28
- Published: 2021-10-28
- Earliest claimed: 2020-04-28
- Assignees: Leela AI, Inc.
- Applicants: Leela AI, Inc.
- Inventors: Henry Bowdoin Minsky; Milan Singh Minsky; Cyrus Shaoul
- Web page: https://patentgrep.com/patent/US20210334671A1

## CPC

- G06F 40/35: Discourse or dialogue representation
- G06F 11/3075: {the data filtering being achieved in order to maintain consistency among the monitored data, e.g. ensuring that the monitored data belong to the same timeframe, to the same system or component}
- G06F 16/2237: {Vectors, bitmaps or matrices}
- G06F 16/2477: {Temporal data queries}
- G06F 16/3329: Natural language query formulation
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/29: {Graphical models, e.g. Bayesian networks}
- G06F 40/279: Recognition of textual entities
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06N 3/092: Reinforcement learning
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N 5/025: Extracting rules from data
- G06N 5/042: {Backward inferencing}
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/44: {Event detection}
- G06F 40/30: Semantic analysis
- G06T 2207/20081: Training; Learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A digital computational learning system and corresponding method plan a series of actions to accomplish tasks. The system learns, automatically, a plurality of actor perceiver predictors (APP) nodes. Each APP node is associated with a context, action, and result. The result is expected to be achieved in response to the action being taken as a function of the context having been satisfied. Each APP node is associated with an action-controller that includes an instance of a planner that includes allied planners. The action-controller is associated with a goal state and employs the allied planners to determine a sequence of actions for reaching the goal state. The allied planners enable the system to plan a series of actions to accomplish complex tasks in a manner that is more robust and resilient relative to current state of the art artificial intelligence based learning systems and methods.

## Claims

### Claim 1

A digital computational learning system for planning actions to accomplish tasks, the digital computational learning system comprising:
a processor configured to learn, automatically, a plurality of actor perceiver predictor (APP) nodes of a knowledge graph, each APP node of the plurality of APP nodes associated with a context, action, and result, the result expected to be achieved in response to the action being taken as a function of the context having been satisfied; and
a memory, the processor further configured to store the knowledge graph in the memory, each APP node associated with an action-controller including an instance of a planner that includes allied planners, the action-controller associated with a goal state and configured to access the knowledge graph and to employ the allied planners to determine a sequence of actions for reaching the goal state by selecting and chaining, dynamically in the memory, at least a portion of APP nodes learned of the plurality of APP nodes learned, the sequence of actions including respective actions of the at least a portion of APP nodes learned, selected, and chained.

### Claim 2 (depends on 1)

The digital computational learning system of claim 1, wherein the allied planners include a graph-based planner and a deep Q-learning (DQL) planner, the graph-based planner configured to provide a partial reward to the DQL planner for progressing toward the goal state, wherein the partial reward is a portion of a total reward for reaching the goal state.

### Claim 3 (depends on 2)

The digital computational learning system of claim 2, wherein the graph-based planner is configured to employ a breadth-first search (BFS) method, wherein the DQL planner is DQL neural network, and wherein the instance of the planner further includes an auxiliary network, the auxiliary network configured to decide whether to trust the graph-based planner or the DQL planner.

### Claim 4 (depends on 1)

The digital computational learning system of claim 1, wherein the context includes a neural network.

### Claim 5 (depends on 1)

The digital computational learning system of claim 1, wherein the sequence of actions determined is a least-cost sequence of actions in order to most efficiently reach the goal state.

### Claim 6 (depends on 1)

The digital computational learning system of claim 1, wherein the processor is a plurality of processors and the memory is a plurality of memories.

### Claim 7 (depends on 1)

The digital computational learning system of claim 1, wherein the processor is further configured to organize and index the plurality of APP nodes.

### Claim 8 (depends on 1)

The digital computational learning system of claim 1, wherein the digital computational learning system further comprises a communications interface and wherein the processor is further configured to employ sensory input to learn, automatically, at least one APP node of the plurality of APP nodes, the sensory input received via the communications interface.

### Claim 9 (depends on 8)

The digital computational learning system of claim 8, wherein the sensory input includes natural language input.

### Claim 10 (depends on 9)

The digital computational learning system of claim 9, wherein the digital computational learning system is deployed in an environment and wherein the natural language input is grounded relative to at least one object in the environment.

### Claim 11 (depends on 10)

The digital computational learning system of claim 10, wherein the environment is a simulated or real-world environment.

### Claim 12 (depends on 1)

The digital computational learning system of claim 1, wherein the digital computational learning system further comprises a communications interface configured to receive natural language input, wherein the natural language input defines a task, and wherein the goal state represents completion of a sub-task for accomplishing the task.

### Claim 13 (depends on 1)

The digital computational learning system of claim 1, wherein the processor is further configured to produce an explanation for why the sequence of actions was determined and wherein the explanation is produced by converting information stored by the digital computational learning system to natural language.

### Claim 14 (depends on 13)

The digital computational learning system of claim 13, wherein the digital computational learning system further comprises a communications interface and wherein the processor is further configured to communicate the explanation via the communications interface in response to a query received via the communications interface.

### Claim 15 (depends on 1)

The digital computational learning system of claim 1, wherein each APP node, of the at least a portion of APP nodes learned, selected, and chained, that precedes a next APP node in a chain of APP nodes, is associated with a corresponding respective result that satisfies a respective context of the next APP in the chain.

### Claim 16

A system for planning actions to accomplish tasks, the system comprising:
a communications interface;
a planning neural network configured to learn, automatically, a plurality of APPs of a knowledge graph and determine a sequence of actions for accomplishing a task by selecting and chaining, dynamically, at least a portion of APPs of the plurality of APPs learned;
a deep queue-learning (DQL) neural network, the planning neural network configured to employ the DQL neural network to accelerate or advance determination of the sequence of actions for accomplishing the task; and
a supervisory system configured to effect a change to the sequence of actions determined by interpreting natural language input and causing the planning neural network to update the selecting and chaining of the at least a portion of APPs of the knowledge graph based on the interpreting, the natural language input received via the communications interface.

### Claim 17 (depends on 16)

The system of claim 16, wherein the planning neural network is further configured to provide at least one partial reward to the DQL neural network.

### Claim 18 (depends on 17)

The system of claim 17, wherein the task includes at least one subtask and the at least one partial reward represents completion of the at least one subtask.

### Claim 19 (depends on 18)

The system of claim 18, wherein completion of the at least one subtask is based on successful execution of at least one action of the sequence of actions determined.

### Claim 20 (depends on 16)

The system of claim 16, wherein the DQL neural network is configured to suggest actions to accelerate or advance the planning neural network's determination of the sequence of actions for accomplishing the task.

### Claim 21 (depends on 16)

The system of claim 16, wherein the graph-based planner is further configured to filter, based on a knowledge database of successful actions, a set of possible actions available to the DQL neural network to train on.

### Claim 22 (depends on 16)

The system of claim 16, wherein the natural language input is associated with an object of an environment within which the system is deployed.

### Claim 23 (depends on 22)

The system of claim 22, wherein the environment is a simulated or real-world environment.

### Claim 24

A computer apparatus comprising:
a controller controlling movement of an automated actor; and
a digital decision-making member coupled to provide directives to the controller, the decision-making member utilizing a machine learning planning graph formed of a plurality of nodes;
different sequences of two or more of the nodes in the graph producing different directives, and
each node having a tuple representing a context, an action, and corresponding result, different nodes having different tuples.

### Claim 25 (depends on 24)

The computer apparatus as claimed in claim 24, wherein the automated actor is one of a robot in a real-world environment, and an agent in a virtual environment.

### Claim 26 (depends on 25)

The computer apparatus as claimed in claim 25, wherein each node employs a neural network.

### Claim 27

A computer-implemented method of learning a virtual environment, the computer-implemented method comprising:
generating a virtual network of interconnected nodes, each node 1) representing a respective symbol and 2) including a respective neural network configured to identify the respective symbol based on observation data;
navigating the virtual environment, based on a goal, to generate the observation data;
identifying, via the respective neural networks, a plurality of symbols from the observation data;
determining, from the observation data, a relation between the plurality of symbols; and
updating the virtual network to indicate the relation as an association between a plurality of the interconnected nodes.

### Claim 28 (depends on 27)

The computer-implemented method of claim 27, wherein the relation indicates a property that is common to each of the plurality of interconnected nodes.

### Claim 29 (depends on 27)

The computer-implemented method of claim 27, further comprising modifying navigation through the environment based on a feedback signal, the feedback signal indicating whether the navigation is progressing towards the goal.

### Claim 30 (depends on 27)

The computer-implemented method of claim 27, wherein the goal indicates a change to be made to an element of the virtual environment and wherein the computer-implemented method further comprises:
identifying a symbol corresponding to the element; and
determining a location of the element in the virtual environment based on at least one association between the symbol and other symbols as indicated by the virtual network of interconnected nodes.

### Claim 31 (depends on 30)

The computer-implemented method of claim 30, further comprising:
navigating the virtual environment to locate the element;
modifying the element in accordance with the goal; and
updating the symbol based on the modification to the element.

### Claim 32

A computer-implemented method for planning actions to accomplish tasks, the computer-implemented method comprising:
learning, automatically by a processor, a plurality of actor perceiver predictor (APP) nodes of a knowledge graph, each APP node of the plurality of APP nodes associated with a context, action, and result, the result expected to be achieved in response to the action being taken as a function of the context having been satisfied;
storing the knowledge graph in a memory;
associating each APP node with an action-controller including an instance of a planner that includes allied planners;
associating the action-controller with a goal state;
accessing the knowledge graph by the action controller; and
employing, by the action controller, the allied planners for determining a sequence of actions for reaching the goal state, the determining including selecting and chaining, dynamically in the memory, at least a portion of APP nodes learned of the plurality of APP nodes learned, the sequence of actions including respective actions of the at least a portion of APP nodes learned, selected, and chained.

### Claim 33 (depends on 32)

The computer-implemented method of claim 32, wherein the allied planners include a graph-based planner and a deep Q-learning (DQL) planner and wherein the computer-implemented method further comprises providing, from the graph-based planner to the DQL planner, a partial reward for progressing toward the goal state, wherein the partial reward is a portion of a total reward for reaching the goal state.

### Claim 34 (depends on 33)

The computer-implemented method of claim 33, wherein the DQL planner is DQL neural network, wherein the instance of the planner further includes an auxiliary network, and wherein the computer-implemented method further comprises:
employing, by the graph-based planner, a breadth-first search (BFS) method; and
deciding, by the auxiliary network, whether to trust the graph-based planner or the DQL planner.

### Claim 35 (depends on 32)

The computer-implemented method of claim 32, further comprising including a neural network in the context.

### Claim 36 (depends on 32)

The computer-implemented method of claim 32, wherein the sequence of actions determined is a least-cost sequence of actions in order to most efficiently reach the goal state.

### Claim 37 (depends on 32)

The computer-implemented method of claim 32, wherein the processor is a plurality of processors and the memory is a plurality of memories.

### Claim 38 (depends on 32)

The computer-implemented method of claim 32, further comprising, by the processor, organizing and indexing the plurality of APP nodes.

### Claim 39 (depends on 32)

The computer-implemented method of claim 32, further comprising:
receiving, by the processor, sensory input via a communications interface; and
learning, automatically by the processor, at least one APP node of the plurality of APP nodes, based on the sensory input received.

### Claim 40 (depends on 39)

The computer-implemented method of claim 39, wherein the sensory input includes natural language input.

### Claim 41 (depends on 40)

The computer-implemented method of claim 40, further comprising grounding the natural language input relative to at least one object in an environment.

### Claim 42 (depends on 41)

The computer-implemented method of claim 41, wherein the environment is a simulated or real-world environment.

### Claim 43 (depends on 32)

The computer-implemented method of claim 32, further comprising receiving natural language input via a communications interface, wherein the natural language input defines a task, and wherein the goal state represents completion of a sub-task for accomplishing the task.

### Claim 44 (depends on 32)

The computer-implemented method of claim 32, further comprising producing, by the processor, an explanation for why the sequence of actions was determined, the producing including converting information stored in the memory to natural language.

### Claim 45 (depends on 44)

The computer-implemented method of claim 44, further comprising receiving a query via the communications interface and communicating the explanation via the communications interface in response to the query received.

### Claim 46 (depends on 32)

The computer-implemented method of claim 32, further comprising associating each APP node, of the at least a portion of APP nodes learned, selected, and chained, that precedes a next APP node in a chain of APP nodes, with a corresponding respective result that satisfies a respective context of the next APP in the chain.

### Claim 47

A computer-implemented method for planning actions to accomplish tasks, the computer-implemented method comprising:
learning, automatically, by a planning neural network, a plurality of APPs of a knowledge graph and determining a sequence of actions for accomplishing a task by selecting and chaining, dynamically, at least a portion of APPs of the plurality of APPs learned;
employing, by the planning neural network, a deep queue-learning (DQL) neural network to accelerate or advance determination of the sequence of actions for accomplishing the task;
receiving natural language input via the communications interface; and
effecting a change to the sequence of actions determined by interpreting the natural language input and causing the planning neural network to update the selecting and chaining of the at least a portion of APPs of the knowledge graph based on the interpreting.

### Claim 48 (depends on 47)

The computer-implemented method of claim 47, further comprising providing, by the planning neural network, at least one partial reward to the DQL neural network.

### Claim 49 (depends on 48)

The computer-implemented method of claim 48, wherein the task includes at least one subtask and the at least one partial reward represents completion of the at least one subtask.

### Claim 50 (depends on 49)

The computer-implemented method of claim 49, wherein completion of the at least one subtask is based on successful execution of at least one action of the sequence of actions determined.

### Claim 51 (depends on 47)

The computer-implemented method of claim 47, further comprising suggesting, by the DQL neural network, actions to accelerate or advance the planning neural network's determination of the sequence of actions for accomplishing the task.

### Claim 52 (depends on 47)

The computer-implemented method of claim 47, further comprising filtering, by the graph-based planner, based on a knowledge database of successful actions, a set of possible actions available to the DQL neural network to train on.

### Claim 53 (depends on 47)

The computer-implemented method of claim 47, wherein the natural language input is associated with an object of an environment within which the system is deployed and wherein the environment is a simulated or real-world environment.

### Claim 54

A computer-implemented method comprising:
controlling, by a controller, movement of an automated actor;
providing, by a digital decision-making member, directives to the controller; and
utilizing, by the decision-making member, a machine learning planning graph formed of a plurality of nodes,
different sequences of two or more of the nodes in the graph producing different directives, and
each node having a tuple representing a context, an action, and corresponding result, different nodes having different tuples.

### Claim 55 (depends on 54)

The computer-implemented method of claim 54, wherein the automated actor is one of a robot in a real-world environment, and an agent in a virtual environment.

### Claim 56 (depends on 54)

The computer-implemented method of claim 54, further comprising employing, by each node, a neural network.
