Detecting anomolies in textual items using cross-entropies

A method for detecting anomalies in textual items comprising:

• receiving a first plurality of textual items by a computing device;
• training a language model using the received first plurality of textual items by the computing device;
• after training the language model, receiving a second plurality of textual items by the computing device;
• for each textual item of the second plurality of content items:
• calculating a cross-entropy for the textual item using the language model by the computing device;
• calculating a perplexity of the language model by the computing device;
• and
• if the calculated perplexity exceeds a threshold, sliding the language model forward using a subset of the textual items of the second plurality by the computing device;
• and
• detecting an anomaly in at least one textual item of the second plurality of textual items by the computing device using the calculated cross-entropies.

A system for detecting anomalies in textual items comprising:

• at least one processor;
• and
• a memory storing instructions that when executed by the at least one processor cause the at least one processor to:
• receive a first plurality of textual items;
• train a language model using the received first plurality of textual items;
• after training the language model, receive a second plurality of textual items;
• calculate a cross-entropy for each textual item in the second plurality of textual items using the language model;
• and
• detect an anomaly in at least one of the textual items of the second plurality of textual items using the calculated cross-entropies.