Byte n-gram embedding model

One or more non-transitory computer-readable media storing instructions that, when executed by a processor, cause a computing device to perform operations comprising:

• extracting a plurality of byte n-grams from a training set of data;
• using the plurality of byte n-grams to train a neural network to determine a probability of occurrence of any byte n-gram, wherein the neural network comprises at least one embedding model layer configured to output an array of values;
• after training the neural network, receiving data to be processed;
• and
• using the neural network to process the received data, wherein processing the received data comprises one or more of:
• generating a hash of the received data;
• generating a hash of a pseudo-random portion of the received data;
• classifying the received data;
• or
• generating a signature of the received data.

A system comprising:

• one or more processors;
• a memory communicatively coupled to the one or more processors;
• a feature extractor stored in the memory and executable by the processor, the feature extractor configured to:
• extract a plurality of byte n-grams from data to be processed;
• extract one or more features from individual byte n-grams of the plurality of byte n-grams;
• a neural network stored in the memory and executable by the processor, the neural network configured to receive the features of a byte n-gram as input, and in response, to output from an embedding model layer of the neural network, an embedding array of values that is calculated based on the features of the byte n-gram.