Analyzing web application behavior to detect malicious requests

A computer implemented method executable on a proxy server, the method for protecting a web application at a host, the method comprising:

• receiving a plurality of requests for functionality of the web application, wherein the plurality of requests include HTTP requests directed to the web application;
• identifying a set of known requests from the plurality of the requests, the set of known requests including known malicious and non-malicious requests, the remaining requests being unknown requests;
• collecting the set of known requests and attributes associated with each known request in the set of known requests, wherein the attributes are determined from each known request by at least preprocessing each known request through a rule based Web Application Firewall;
• wherein collecting the attributes from each known HTTP request includes deriving static attributes directly from the known HTTP request that are attributes including an HTTP method, an HTTP version, a uniform resource indicator, a hostname, and a request body size;
• transmitting the collected set of known requests and their associated static attributes to offload generation of a model;
• generating a first set of request vectors including at least one vector for each known malicious request based on the static attributes of the known malicious request;
• generating a second set of request vectors including at least one vector for each known non-malicious request based on the static attributes of the known non-malicious request;
• wherein the model is generated by at least machine learning including inputting the first set of request vectors and the second set of request vectors to train the model to identify malicious requests and non-malicious requests;
• receiving the model configured by machine learning to classify unknown requests for functionality of the web application;
• and
• in response to receiving an HTTP unknown request in the plurality of requests and, for each unknown request:
• determining static attributes directly from the unknown HTTP request including at least an HTTP method, an HTTP version, a uniform resource indicator, a hostname, and a request body size;
• determining additional attributes associated with the unknown HTTP request comprising a challenge indication by:
• (i) challenging a client device that originated the unknown HTTP request;
• and
• (ii) setting the challenge indication based on whether the client device that originated the unknown request passed or failed the challenge;
• providing the static attributes and the additional attributes of the unknown HTTP request to the model for determining a classification for the unknown HTTP request wherein the model classifies the unknown HTTP request as malicious or non-malicious based on at least comparing the first set of request vectors and the second set of request vectors to the static attributes of the unknown HTTP request and based on the challenge indication;
• and
• denying or permitting a client that originated the unknown request access to the web application based on the classification.

A system for protecting a web application at a host, the system comprising

• a proxy server configured to:
• receive a plurality of requests for functionality of the web application, wherein the plurality of requests include HTTP requests directed to the web application;
• identify a set of known requests from the plurality of the requests, the set of known requests including known malicious and non-malicious requests, the remaining requests being unknown requests;
• determining attributes associated with each of the known requests comprising preprocessing each known request through a rule based Web Application Firewall;
• wherein determining the attributes from each known HTTP request includes deriving static attributes directly from the known HTTP request that are attributes including an HTTP method, an HTTP version, a uniform resource indicator, a hostname, and a request body size;
• transmit the set of known requests and their associated static attributes to offload generation of a model, and
• in response to receiving an unknown HTTP request in the plurality of requests and, for each unknown HTTP request:
• transmit a query for a classification of the unknown request, the query including attributes associated with the unknown request, wherein the attributes are determined by:
• determining static attributes directly from the unknown HTTP request including at least an HTTP method, an HTTP version, a uniform resource indicator, a hostname, and a request body size;
• determining additional attributes associated with the unknown HTTP request comprising a challenge indication by:
• (i) challenging the client device that originated the unknown HTTP request;
• and
• (ii) setting the challenge indication based on whether the client device that originated the unknown request passed or failed the challenge;
• in response to receiving the classification of the unknown HTTP request, denying or permitting a client that originated the unknown request access to the web application based on the classification;
• a model generator configured to:
• generate a first set of request vectors including at least one vector for each known malicious request based on at least the static attributes of the known malicious request;
• generate a second set of request vectors including at least one vector for each known non-malicious request based on at least the static attributes of the known non-malicious request;
• ingest the first set of request vectors and the second set of request vectors from the set of known requests and their static attributes collected from the proxy server, and
• generate, based on the first set of request vectors and the second set of request vectors, a model to classify unknown requests for functionality of the web application, wherein the model is generated by at least machine learning including inputting the first set of request vectors and the second set of request vectors to train the model to identify malicious requests and non-malicious requests;
• and
• a prediction service configured to:
• load the model generated by the model generator,
• receive the query for the classification of the unknown request, and
• transmit the classification for the unknown request to the proxy server, wherein the model classifies the unknown request as malicious or non-malicious based on at least comparing the first set of request vectors and the second set of request vectors to the static attributes of the unknown HTTP request and based on the challenge indication.