# US 9,519,778 B2 · Systems, methods, and media for outputting a dataset based upon anomaly detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,519,778 B2 - Document ID: US9519778B2 - Status: Granted - Kind: B2 - Application number: 13/769,774 - Filed: 2013-02-18 - Published: 2014-03-20 - Granted: 2016-12-13 - Earliest claimed: 2006-02-28 - Assignees: The Trustees of Columbia University in the City of New York - Applicants: The Trustees of Columbia University in the City of New York - Inventors: Salvatore J Stolfo; Ke Wang; Janak Parekh - Web page: https://patentgrep.com/patent/US9519778B2 ## CPC - G06F 21/56: Computer malware detection or handling, e.g. anti-virus arrangements - G06F 21/564: {by virus signature recognition} - H04L 63/1416: {Event detection, e.g. attack signature detection} - H04L 63/1425: {Traffic logging, e.g. anomaly detection} - G06F 2221/034: Test or assess a computer or a system - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Systems, methods, and media for outputting a dataset based upon anomaly detection are provided. In some embodiments, methods for outputting a dataset based upon anomaly detection: receive a training dataset having a plurality of n-grams, which plurality includes a first plurality of distinct training n-grams each being a first size; compute a first plurality of appearance frequencies, each for a corresponding one of the first plurality of distinct training n-grams; receive an input dataset including first input n-grams each being the first size; define a first window in the input dataset; identify as being first matching n-grams the first input n-grams in the first window that correspond to the first plurality of distinct training n-grams; compute a first anomaly detection score for the input dataset using the first matching n-grams and the first plurality of appearance frequencies; and output the input dataset based on the first anomaly detection score. ## Claims ### Claim 1 A method for outputting a dataset based upon anomaly detection, the method comprising: receiving, using a hardware processor, an input dataset; determining, using the hardware processor, an anomaly detection score that is indicative of the presence of anomalous n-grams in the input dataset by applying a content anomaly detection model to the received input data, wherein the content anomaly detection model is trained with data that includes training n-grams and wherein the anomaly detection score is based on a comparison of n-grams in the input dataset, which each represent a sequence of n bytes, with the training n-grams; and causing, using the hardware processor, the input dataset to be outputted based on the anomaly detection score. ### Claim 2 (depends on 1) The method of claim 1, wherein the content anomaly detection model is a frequency distribution-based detection model that determines a plurality of appearance frequencies, and wherein each of the plurality of appearance frequencies corresponds to one of the training n-grams. ### Claim 3 (depends on 2) The method of claim 2, wherein the training n-grams include a first plurality of distinct training n-grams and a second plurality of distinct training n-grams. ### Claim 4 (depends on 3) The method of claim 3, further comprising: determining a first plurality of appearance frequencies corresponding to the first plurality of distinct training n-grams; and determining a second plurality of appearance frequencies corresponding to the second plurality of distinct training n-grams; and determining a first anomaly detection score based on the first plurality of appearance frequencies and a second anomaly detection score based on the second plurality of appearance frequencies. ### Claim 5 (depends on 4) The method of claim 4, further comprising: determining which of the first anomaly detection score and the second anomaly detection score is higher; and outputting the input dataset based on the anomaly detection score that is higher. ### Claim 6 (depends on 1) The method of claim 1, wherein the content anomaly detection model is a binary-based detection model that determines a number of distinct n-grams in the input dataset and a total number of n-grams contained in the input dataset. ### Claim 7 A system for outputting a dataset based upon anomaly detection, the system comprising: a hardware processor that: receives an input dataset; determines an anomaly detection score that is indicative of the presence of anomalous n-grams in the input dataset by applying a content anomaly detection model to the received input data, wherein the content anomaly detection model is trained with data that includes training n-grams and wherein the anomaly detection score is based on a comparison of n-grams in the input dataset, which each represent a sequence of n bytes, with the training n-grams; and causes the input dataset to be outputted based on the anomaly detection score. ### Claim 8 (depends on 7) The system of claim 7, wherein the content anomaly detection model is a frequency distribution-based detection model that determines a plurality of appearance frequencies, and wherein each of the plurality of appearance frequencies corresponds to one of the training n-grams. ### Claim 9 (depends on 8) The system of claim 8, wherein the training n-grams include a first plurality of distinct training n-grams and a second plurality of distinct training n-grams. ### Claim 10 (depends on 9) The system of claim 9, wherein the system is further configured to: determine a first plurality of appearance frequencies corresponding to the first plurality of distinct training n-grams; and determine a second plurality of appearance frequencies corresponding to the second plurality of distinct training n-grams; and determine a first anomaly detection score based on the first plurality of appearance frequencies and a second anomaly detection score based on the second plurality of appearance frequencies. ### Claim 11 (depends on 10) The system of claim 10, wherein the system is further configured to: determine which of the first anomaly detection score and the second anomaly detection score is higher; and output the input dataset based on the anomaly detection score that is higher. ### Claim 12 (depends on 7) The system of claim 7, wherein the content anomaly detection model is a binary-based detection model that determines a number of distinct n-grams in the input dataset and a total number of n-grams contained in the input dataset. ### Claim 13 A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for outputting a dataset based upon anomaly detection, the method comprising: receiving an input dataset; determining an anomaly detection score that is indicative of the presence of anomalous n-grams in the input dataset by applying a content anomaly detection model to the received input data, wherein the content anomaly detection model is trained with data that includes training n-grams and wherein the anomaly detection score is based on a comparison of n-grams in the input dataset, which each represent a sequence of n bytes, with the training n-grams; and causing the input dataset to be outputted based on the anomaly detection score. ### Claim 14 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the content anomaly detection model is a frequency distribution-based detection model that determines a plurality of appearance frequencies, and wherein each of the plurality of appearance frequencies corresponds to one of the training n-grams. ### Claim 15 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the training n-grams include a first plurality of distinct training n-grams and a second plurality of distinct training n-grams. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further comprises: determining a first plurality of appearance frequencies corresponding to the first plurality of distinct training n-grams; and determining a second plurality of appearance frequencies corresponding to the second plurality of distinct training n-grams; and determining a first anomaly detection score based on the first plurality of appearance frequencies and a second anomaly detection score based on the second plurality of appearance frequencies. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the method further comprises: determining which of the first anomaly detection score and the second anomaly detection score is higher; and outputting the input dataset based on the anomaly detection score that is higher. ### Claim 18 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the content anomaly detection model is a binary-based detection model that determines a number of distinct n-grams in the input dataset and a total number of n-grams contained in the input dataset.