Reinforcement-based system and method for detecting system vulnerabilities

A system for detecting vulnerabilities for a computer application, the system comprising:

• a processor;
• a non-transitory computer memory storing instructions and data representing a machine learning model, wherein the instructions when executed, configures the processor to:
• use the machine learning model to generate a payload to test vulnerabilities in an application, the machine learning model being trained based on an action space comprising one or more input data, wherein a machine learning agent is configured by the processor to:
• select one or more input data from the action space to form an action data;
• electronically provide the action data as an input to the application;
• monitor utilization of one or more system resources in response to the action data being inputted to the application;
• determine a score of utilization based on a result of the monitoring;
• determine a corresponding reward score for the action data based on the score of utilization;
• and
• identify one or more of the action data to form a pool of candidate data for the application, wherein each of action data has a corresponding reward score over a specific threshold.