# US 2021/0174240 A1 · Data augmented training of reinforcement learning software agent

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174240 A1
- Document ID: US20210174240A1
- Status: Published application
- Kind: A1
- Application number: 16/704,395
- Filed: 2019-12-05
- Published: 2021-06-10
- Earliest claimed: 2019-12-05
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Tathagata Chakraborti; Kartik Talamadupula; Kshitij Fadnis; Biplav Srivastava; Murray S. Campbell
- Web page: https://patentgrep.com/patent/US20210174240A1

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06F 16/24578: {using ranking}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/08: Learning methods
- G06N 3/092: Reinforcement learning
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06N 5/041: {Abduction}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques are provided for reinforcement learning software agents enhanced by external data. A reinforcement learning model supporting the software agent may be trained based on information obtained from one or more knowledge stores, such as online forums. The trained reinforcement learning model may be tested in an environment with limited connectivity to an external environment to meet performance criteria. The reinforcement learning software agent may be deployed with the tested and trained reinforcement learning model within an environment to autonomously perform actions to process requests.

## Claims

### Claim 1

A method of using data from a knowledge store to configure a reinforcement learning software agent, the method comprising:
receiving, by a computer, access to the knowledge store regarding a topic for software agent support;
using, by the computer, information from the knowledge store for training a reinforcement learning model supporting the reinforcement learning software agent;
testing, by the computer, the trained reinforcement learning model in a testing environment, the testing environment having limited connectivity to an external environment; and
deploying the reinforcement learning software agent with the tested and trained reinforcement learning model within an environment to autonomously perform actions to process requests.

### Claim 2 (depends on 1)

The method of claim 1, wherein the information from the knowledge store is in a semi-structured format and includes questions with one or more corresponding answers.

### Claim 3 (depends on 2)

The method of claim 2, wherein the knowledge store comprises information generated by a plurality of users, and wherein at least part of the information is curated by the plurality of users.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
providing access to a domain simulator to execute actions by the reinforcement learning software agent in the testing environment.

### Claim 5 (depends on 1)

The method of claim 1, wherein the knowledge store includes features that are used by the reinforcement learning model to rank the information provided in the knowledge store.

### Claim 6 (depends on 5)

The method of claim 5, wherein the features include one or more of upvotes, downvotes, an author name, an author title, or an author status.

### Claim 7 (depends on 1)

The method of claim 1, wherein the reinforcement learning model ranks the information in the knowledge store based on user preferences.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
generating a query;
matching the query to a state in a policy generated by the reinforcement learning model, wherein each state has a corresponding action; and
executing by the reinforcement learning software agent the action returned from the matching policy.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
receiving a reward based on executing the action by the reinforcement learning software agent, when the action resolves the query.

### Claim 10 (depends on 1)

The method of claim 1, wherein a planning decision agent and a random software agent are used to validate learned actions by the reinforcement learning software agent.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
updating a policy generated by the reinforcement learning model, at a fixed or dynamic frequency, to include questions and answers added to the knowledge store.

### Claim 12 (depends on 1)

The method of claim 1, wherein the reinforcement learning software agent deployed with the trained reinforcement learning model reaches a desired goal more efficiently than a reinforcement learning software agent without a trained reinforcement learning model.

### Claim 13

A computer system for adapting processing of expressions by a command-line interface, the computer system comprising:
one or more computer processors;
one or more computer readable storage media;
program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising instructions to:
receive, by a computer, access to the knowledge store regarding a topic for software agent support;
use, by the computer, information from the knowledge store to train a reinforcement learning model supporting the reinforcement learning software agent;
test, by the computer, the trained reinforcement learning model in a testing environment, the testing environment having limited connectivity to an external environment; and
deploy the reinforcement learning software agent with the tested and trained reinforcement learning model within an environment to autonomously perform actions to process requests.

### Claim 14 (depends on 13)

The computer system of claim 13, wherein the information from the knowledge store is in a semi-structured format and includes questions with one or more corresponding answers.

### Claim 15 (depends on 14)

The computer system of claim 14, wherein the knowledge store comprises information generated by a plurality of users, and wherein at least part of the information is curated by the plurality of users.

### Claim 16 (depends on 13)

The computer system of claim 13, wherein the program instructions further comprise instructions to:
provide access to a domain simulator to execute actions by the reinforcement learning software agent in the testing environment.

### Claim 17 (depends on 13)

The computer system of claim 13, wherein the knowledge store includes features that are used by the reinforcement learning model to rank the information provided in the knowledge store, wherein the features include one or more of upvotes, downvotes, an author name, an author title, or an author status.

### Claim 18 (depends on 13)

The computer system of claim 13, wherein the program instructions further comprise instructions to:
generate a query;
match the query to a state in a policy generated by the reinforcement learning model, wherein each state has a corresponding action;
execute by the reinforcement learning software agent the action returned from the matching policy; and
receive a reward based on executing the action by the reinforcement learning software agent, when the action resolves the query.

### Claim 19 (depends on 13)

The computer system of claim 13, wherein the reinforcement learning software agent deployed with the trained reinforcement learning model reaches a desired goal more efficiently than a reinforcement learning software agent without a trained reinforcement learning model.

### Claim 20

A computer program product for adapting processing of expressions by a command-line interface, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
receive, by a computer, access to the knowledge store regarding a topic for software agent support;
use, by the computer, information from the knowledge store to train a reinforcement learning model supporting the reinforcement learning software agent;
test, by the computer, the trained reinforcement learning model in a testing environment, the testing environment having limited connectivity to an external environment; and
deploy the reinforcement learning software agent with the tested and trained reinforcement learning model within an environment to autonomously perform actions to process requests.

### Claim 21 (depends on 20)

The computer program product of claim 20, wherein the information from the knowledge store is in a semi-structured format and includes questions with one or more corresponding answers.

### Claim 22 (depends on 21)

The computer program product of claim 21, wherein the knowledge store comprises information generated by a plurality of users, and wherein at least part of the information is curated by the plurality of users.

### Claim 23 (depends on 20)

The computer program product of claim 20, wherein the program instructions further comprise instructions to:
provide access to a domain simulator to execute actions by the reinforcement learning software agent in the testing environment.

### Claim 24 (depends on 20)

The computer program product of claim 20, wherein the program instructions further comprise instructions to:
generate a query;
match the query to a state in a policy generated by the reinforcement learning model, wherein each state has a corresponding action;
execute by the reinforcement learning software agent the action returned from the matching policy; and
receive a reward based on executing the action by the reinforcement learning software agent, when the action resolves the query.

### Claim 25 (depends on 20)

The computer program product of claim 20, wherein the reinforcement learning software agent deployed with the trained reinforcement learning model reaches a desired goal more efficiently than a reinforcement learning software agent without a trained reinforcement learning model.
