# US 10,872,157 B2 · Reinforcement-based system and method for detecting system vulnerabilities > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,157 B2 - Document ID: US10872157B2 - Status: Granted - Kind: B2 - Application number: 16/230,804 - Filed: 2018-12-21 - Published: 2019-06-27 - Granted: 2020-12-22 - Earliest claimed: 2017-12-22 - Assignees: Royal Bank of Canada - Applicants: Royal Bank of Canada - Inventors: Cory Fong - Web page: https://patentgrep.com/patent/US10872157B2 ## CPC - G06F 21/577: {Assessing vulnerabilities and evaluating computer system security} - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 20/00: Machine learning - G06F 2221/033: Test or assess software - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A system and method for testing system vulnerabilities are provided. The method may include: training, by a processor, a machine learning model and agent to generate a payload to test vulnerabilities in the application by: selecting one or more input data from the action space to form an action data; electronically providing the action data as an input to the application; monitoring utilization of one or more system resources in response to the action data being inputted to the application; determining a score of utilization based on a result of the monitoring; determining a corresponding reward score for the action data based on the score of utilization; and identifying one or more of the action data to form a pool of candidate data for the application. ## Claims ### Claim 1 A system for detecting vulnerabilities for a computer application, the system comprising: a processor; a non-transitory computer memory storing instructions and data representing a machine learning model, wherein the instructions when executed, configures the processor to: use the machine learning model to generate a payload to test vulnerabilities in an application, the machine learning model being trained based on an action space comprising one or more input data, wherein a machine learning agent is configured by the processor to: select one or more input data from the action space to form an action data; electronically provide the action data as an input to the application; monitor utilization of one or more system resources in response to the action data being inputted to the application; determine a score of utilization based on a result of the monitoring; determine a corresponding reward score for the action data based on the score of utilization; and identify one or more of the action data to form a pool of candidate data for the application, wherein each of action data has a corresponding reward score over a specific threshold. ### Claim 2 (depends on 1) The system of claim 1, wherein the processor is configured to iteratively update the machine learning model based on at least one of the score of utilization and the corresponding reward score. ### Claim 3 (depends on 1) The system of claim 1, wherein the processor is configured to: select one or more input data from the action space to form a vulnerability test input for the application based on the identified candidate data; and electronically provide the vulnerability test input to the application to test the application. ### Claim 4 (depends on 1) The system of claim 1, wherein the one or more input data comprises at least one of: character data, command data and user action data. ### Claim 5 (depends on 1) The system of claim 1, wherein the one or more system resources comprises at least one processor, memory, or register. ### Claim 6 (depends on 1) The system of claim 1, wherein the processor is configured to further select one or more input data from the action space to form a new action data based on one or more of the reward scores. ### Claim 7 (depends on 4) The system of claim 4, wherein the action data comprises a text string. ### Claim 8 (depends on 1) The system of claim 1, wherein each application has a corresponding action space. ### Claim 9 (depends on 1) The system of claim 1, wherein determining the score of utilization comprises: determining a base usage of the one or more system resources for the application; determining a second usage of the one or more system resources for the application with the action data; and determining the score of utilization based on the base usage and the second usage of the one or more system resources for the application. ### Claim 10 (depends on 9) The system of claim 9, wherein the base usage of the one or more system resources is determined based on monitoring the usage of the one or more system resources when the application is running under normal circumstances without the action data. ### Claim 11 (depends on 9) The system of claim 9, wherein determining the score of utilization based on the base usage and the second usage of the one or more system resources for the application comprises generating the score of utilization by comparing the second usage to the base usage of the one or more system resources. ### Claim 12 A computer-implemented method for detecting vulnerabilities for a computer application, the method comprising: training, by a processor, a machine learning model to generate a payload to test vulnerabilities in the application, the machine learning model being trained based on an action space comprising one or more input data; and executing, by the processor, instructions to cause a machine learning agent to: select one or more input data from the action space to form an action data; electronically provide the action data as an input to the application; monitor utilization of one or more system resources in response to the action data being inputted to the application; determine a score of utilization based on a result of the monitoring; determine a corresponding reward score for the action data based on the score of utilization; and identify one or more of the action data to form a pool of candidate data for the application, wherein each of action data has a corresponding reward score over a specific threshold. ### Claim 13 (depends on 12) The method of claim 12, comprising iteratively updating the machine learning model based on at least one of the score of utilization and the corresponding reward score. ### Claim 14 (depends on 12) The method of claim 12, comprising: selecting one or more input data from the action space to form a vulnerability test input for the application based on the identified candidate data; and electronically providing the vulnerability test input to the application to test the application. ### Claim 15 (depends on 12) The method of claim 12, wherein the one or more input data comprises at least one of: character data, command data and user action data. ### Claim 16 (depends on 12) The method of claim 12, wherein the one or more system resources comprises at least one processor, memory, or register. ### Claim 17 (depends on 12) The method of claim 12, wherein the processor is configured to further select one or more input data from the action space to form a new action data based on one or more of the reward scores. ### Claim 18 (depends on 12) The method of claim 12, wherein determining the score of utilization comprises: determining a base usage of the one or more system resources for the application; determining a second usage of the one or more system resources for the application with the action data; and determining the score of utilization based on the base usage and the second usage of the one or more system resources for the application. ### Claim 19 (depends on 18) The method of claim 18, wherein the base usage of the one or more system resources is determined based on monitoring the usage of the one or more system resources when the application is running under normal circumstances without the action data. ### Claim 20 (depends on 18) The method of claim 18, wherein determining the score of utilization based on the base usage and the second usage of the one or more system resources for the application comprises generating the score of utilization by comparing the second usage to the base usage of the one or more system resources.