# US 11,443,130 B2 · Making a failure scenario using adversarial reinforcement learning background > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,130 B2 - Document ID: US11443130B2 - Status: Granted - Kind: B2 - Application number: 16/557,453 - Filed: 2019-08-30 - Published: 2021-03-04 - Granted: 2022-09-13 - Earliest claimed: 2019-08-30 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Akifumi Wachi - Web page: https://patentgrep.com/patent/US11443130B2 ## CPC - G06N 3/08: Learning methods - B60W 50/0097: {Predicting future conditions} - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/22: Matching criteria, e.g. proximity measures - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/092: Reinforcement learning - G06N 3/094: Adversarial learning - G06N 20/00: Machine learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Making failure scenarios using adversarial reinforcement learning is performed by storing, in a first storage, a variety of first experiences of failures of a player agent due to an adversarial agent, and performing a simulation of an environment including the player agent and the adversarial agent. It also includes calculating a similarity of a second experience of a failure of the player agent in the simulation and each of the variety of first experiences in the first storage, and updating the first storage by adding the second experience as a new first experience of the variety of first experiences in response to the similarity being less than a threshold. Additionally, the use of adversarial reinforcement learning can include training the adversarial agent by using at least one of the plurality of first experiences in the first storage to generate an adversarial agent having diverse experiences. ## Claims ### Claim 1 A computer-implemented method comprising: storing, in a first storage, a plurality of first experiences of failures of a player agent due to an adversarial agent; performing a simulation of an environment including the player agent and the adversarial agent; calculating a similarity of a second experience of a failure of the player agent in the simulation and each of the plurality of first experiences in the first storage; updating the first storage by adding the second experience as a new first experience of the plurality of first experiences in response to the similarity being less than a threshold; and training the adversarial agent by using at least one of the plurality of first experiences in the first storage to generate an adversarial agent having diverse experiences. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising outputting the plurality of first experiences of failures of the player agent collected in the first storage. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the calculating the similarity includes: extracting a feature of the second experience and a feature of each of the plurality of first experiences; and calculating the similarity of the second experience and each of the plurality of first experiences based on the feature of the second experience and the feature of each of the plurality of first experiences. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the performing includes executing a rule-based algorithm of the player agent and reflecting the result of the execution of the rule-based algorithm in the simulation. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the player agent is an autonomous driving vehicle algorithm and the adversarial agent is a machine learning model of a simulated vehicle. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising storing, in a second storage, a plurality of third experiences, each third experience being a success of the player agent or a failure of the player agent in which the similarity is equal to or larger than the threshold, wherein the training includes training the adversarial agent by using at least one of the plurality of first experiences in the first storage and at least one of the plurality of third experiences in the second storage. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the training includes training each adversarial agent of a plurality of the adversarial agents by using a set of first experiences randomly selected from the plurality of first experiences in the first storage and a set of third experiences randomly selected from the plurality of third experiences in the second storage, wherein the random selections are performed for each adversarial agent. ### Claim 8 (depends on 6) The computer-implemented method of claim 6, further comprising updating the second storage by adding the second experience as a new third experience in response to the similarity being equal to or larger than the threshold. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the calculating includes assigning higher adversarial reward to the second experience if the similarity is less than the threshold and lower adversarial reward to the second experience if the similarity is equal to or higher than the threshold. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein the adversarial agent receives a personal reward and the adversarial reward, the personal reward being a reward for achieving a purpose of the adversarial agent regardless of whether or not the adversarial agent disturbs the player agent. ### Claim 11 A computer program product including one or more computer readable storage mediums collectively storing program instructions that are executable by a processor or programmable circuitry to cause the processor or programmable circuitry to perform operations comprising: storing, in a first storage, a plurality of first experiences of failures of a player agent due to an adversarial agent; performing a simulation of an environment including the player agent and the adversarial agent; calculating a similarity of a second experience of a failure of the player agent in the simulation and each of the plurality of first experiences in the first storage; updating the first storage by adding the second experience as a new first experience of the plurality of first experiences in response to the similarity being less than a threshold; and training the adversarial agent by using at least one of the plurality of first experiences in the first storage to generate an adversarial agent having diverse experiences. ### Claim 12 (depends on 11) The computer program product of claim 11, wherein the operations further comprise outputting a plurality of first experiences of failures of the player agent collected in the first storage. ### Claim 13 (depends on 11) The computer program product of claim 11, wherein the calculating the similarity includes: extracting a feature of the second experience and a feature of each of the plurality of first experiences; and calculating the similarity of the second experience and each of the plurality of first experiences based on the feature of the second experience and the feature of each of the plurality of first experiences. ### Claim 14 (depends on 11) The computer program product of claim 11, wherein the player agent is an autonomous driving vehicle algorithm and the adversarial agent is a machine learning model of a simulated vehicle. ### Claim 15 (depends on 11) The computer program product of claim 11, wherein the operations further comprise storing, in a second storage, a plurality of third experiences, each third experience being a success of the player agent or a failure of the player agent in which the similarity is equal to or larger than the threshold, wherein the training includes training the adversarial agent by using at least one of the plurality of first experiences in the first storage and at least one of the plurality of third experiences in the second storage. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the training includes training each adversarial agent of a plurality of the adversarial agents by using a set of first experiences randomly selected from the plurality of first experiences in the first storage and a set of third experiences randomly selected from the plurality of third experiences in the second storage, wherein the random selections are performed for each adversarial agent. ### Claim 17 (depends on 15) The computer program product of claim 15, wherein the operations further comprising updating the second storage by adding the second experience as a new third experience in response to the similarity being equal to or larger than the threshold. ### Claim 18 An apparatus comprising: a processor or a programmable circuitry; and one or more computer readable mediums collectively including instructions that, when executed by the processor or the programmable circuitry, cause the processor or the programmable circuitry to: store, in a first storage, a plurality of first experiences of failures of a player agent due to an adversarial agent; perform a simulation of an environment including the player agent and the adversarial agent; calculate a similarity of a second experience of a failure of the player agent in the simulation and each of the plurality of first experiences in the first storage; update the first storage by adding the second experience as a new first experience of the plurality of first experiences in response to the similarity being less than a threshold; and train the adversarial agent by using at least one of the plurality of first experiences in the first storage to generate an adversarial agent having diverse experiences. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein the instructions further cause the processor or the programmable circuitry to output a plurality of first experiences of failures of the player agent collected in the first storage. ### Claim 20 (depends on 18) The apparatus of claim 18, wherein the calculating the similarity includes: extracting a feature of the second experience and a feature of each of the plurality of first experiences; and calculating the similarity of the second experience and each of the plurality of first experiences based on the feature of the second experience and the feature of each of the plurality of first experiences. ### Claim 21 (depends on 18) The apparatus of claim 18, wherein the player agent is an autonomous driving vehicle algorithm and the adversarial agent is a machine learning model of a simulated vehicle. ### Claim 22 (depends on 18) The apparatus of claim 18, wherein the instructions further cause the processor or the programmable circuitry to store, in a second storage, a plurality of third experiences, each third experience being a success of the player agent or a failure of the player agent in which the similarity is equal to or larger than the threshold, wherein the training includes training the adversarial agent by using at least one of the plurality of first experiences in the first storage and at least one of the plurality of third experiences in the second storage. ### Claim 23 (depends on 22) The apparatus of claim 22, wherein the training includes training each adversarial agent of a plurality of the adversarial agents by using a set of first experiences randomly selected from the plurality of first experiences in the first storage and a set of third experiences randomly selected from the plurality of third experiences in the second storage, wherein the random selections are performed for each adversarial agent. ### Claim 24 (depends on 22) The apparatus of claim 22, wherein the instructions further cause the processor or the programmable circuitry to update the second storage by adding the second experience as a new third experience in response to the similarity being equal to or larger than the threshold. ### Claim 25 An apparatus comprising: a first storage configured to store a plurality of first experiences of failures of a player agent due to an adversarial agent; a simulating section configured to perform a simulation of an environment including the player agent and the adversarial agent; a calculating section configured to calculate a similarity of a second experience of a failure of the player agent in the simulation and each of the plurality of first experiences in the first storage; an updating section configured to update the first storage by adding the second experience as a new first experience of the plurality of first experiences in response to the similarity being less than a threshold; and a training section configured to train the adversarial agent by using at least one of the plurality of first experiences in the first storage to generate an adversarial agent having diverse experiences.