Flexible deterministic finite automata (DFA) tokenizer for AI-based malicious traffic detection

A non-transitory machine-readable medium having instructions comprising software modules stored thereon comprising instructions configured to be executed by one or more processors in one or more computer system, the software modules including:

• a Deterministic Finite Automata (DFA) compiler, configured to process a profile comprising a plurality of strings and associated rules and generate a DFA transition table;
• a DFA engine, configured to receive an input string and generate a token sequence using the DFA transition table;
• and
• a feature extraction engine, configured to extract features from the token sequences generated by the DFA engine and convert the extracted features into feature vectors,
• wherein the feature vectors are configured to be used to train an artificial intelligence (AI) model to detect whether inputs comprising strings are malicious or benign.

A Web Application Firewall (WAF) comprising:

• one or more processors;
• memory, operatively coupled to the one or more processors;
• at least one network interface having one or more ports, operatively coupled to the processor;
• and
• software instructions comprising a Deterministic Finite Automata (DFA) tokenizer, a feature extraction engine, and a machine learning model configured to perform binary classification,
• where the WAF is enabled, via execution of the software instructions to:
• extract strings from inputs received at least one of the one or more ports of the network interface;
• convert the strings to token sequences using the (DFA) tokenizer;
• process the token sequences with the feature extraction engine to generate feature vectors;
• and
• classify the feature vectors using the machine learning model to determine whether the inputs are malicious or benign.