# US 2014/0196147 A1 · Apparatus method and medium for detecting payload anomaly using n-gram distribution of normal data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0196147 A1 - Document ID: US20140196147A1 - Status: Published application - Kind: A1 - Application number: 14/146,533 - Filed: 2014-01-02 - Published: 2014-07-10 - Earliest claimed: 2003-11-12 - Assignees: The Trustees of Columbia University in the City of New York - Applicants: The Trustees of Columbia University In the City of New York - Inventors: Salvatore J. Stolfo; Ke Wang - Web page: https://patentgrep.com/patent/US20140196147A1 ## CPC - H04L 63/1425: {Traffic logging, e.g. anomaly detection} - G06F 21/55: Detecting local intrusion or implementing counter-measures - G06F 21/552: {involving long-term monitoring or reporting} - G06F 21/554: {involving event detection and direct action} - G06F 21/56: Computer malware detection or handling, e.g. anti-virus arrangements - G06F 21/562: {Static detection} - G06F 21/563: {by source code analysis} - G06F 21/564: {by virus signature recognition} - H04L 43/00: Arrangements for monitoring or testing data switching networks - H04L 43/0876: Network utilisation, e.g. volume of load or congestion level - H04L 63/0218: {Distributed architectures, e.g. distributed firewalls} - H04L 63/0245: {Filtering by information in the payload} - H04L 63/0263: {Rule management} - H04L 63/029: {Firewall traversal, e.g. tunnelling or, creating pinholes} - H04L 63/145: {the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms} - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method, apparatus and medium are provided for detecting anomalous payloads transmitted through a network. The system receives payloads within the network and determines a length for data contained in each payload. A statistical distribution is generated for data contained in each payload received within the network, and compared to a selected model distribution representative of normal payloads transmitted through the network. The model payload can be selected such that it has a predetermined length range that encompasses the length for data contained in the received payload. Anomalous payloads are then identified based on differences detected between the statistical distribution of received payloads and the model distribution. The system can also provide for automatic training and incremental updating of models. ## Claims ### Claim at position 1 1-67. (canceled) ### Claim 68 A method for verifying a file type, the method comprising: receiving, using a hardware processor, a file identified as corresponding to a first file type from a first source; generating a byte value statistical distribution of the data included in the file received from the first source; selecting a model byte value statistical distribution representative of the first file type from model byte value statistical distributions representative of a plurality of file types; determining a distance metric between the byte value statistical distribution of the data included in the file and the selected model byte value statistical distribution; and verifying that a file type of the received file is the first file type based on a comparison of the distance metric to a distance metric threshold indicating that the file type of received file is the first file type. ### Claim 69 (depends on 68) The method of claim 68, further comprising generating an alert that the file type of the received file is not the first file type based on the comparison of the distance metric to the distance metric threshold indicating that the received file is not of the first file type. ### Claim 70 (depends on 68) The method of claim 68, further comprising inhibiting the file from being executed based on the comparison of the distance metric to the distance metric threshold indicating that the received file is not of the first file type. ### Claim 71 (depends on 68) The method of claim 68, further comprising causing the file to be tested to determine whether the file includes a malicious program based on the comparison of the distance metric to the distance metric threshold indicating that the received file is not of the first file type. ### Claim 72 (depends on 68) The method of claim 68, wherein the byte value statistical distribution of the file is a byte value distribution of the average frequency and variance of data contained in the file, and the selected model distribution is a byte value distribution of the average frequency and variance representative of files of the first file type. ### Claim 73 (depends on 68) The method of claim 68, wherein the byte value statistical distribution of the file is a byte value distribution of the data contained in the file, and the selected model distribution is a byte value distribution of representative of files of the first file type. ### Claim 74 (depends on 73) The method of claim 73, wherein the byte value distribution of the file is a byte frequency count of data contained in the file, and the selected model distribution is a byte frequency count for files of the first file type. ### Claim 75 (depends on 73) The method of claim 73, wherein the byte value distribution of the file is a rank ordered byte frequency count of data contained in the file, and the selected model distribution is a rank ordered byte frequency count for files of the first file type. ### Claim 76 (depends on 68) The method of claim 68, wherein the distance metric is calculated based on a Mahalanobis distance between the byte value statistical distribution of the data included in the file and the selected model byte value statistical distribution. ### Claim 77 A system for verifying a file type, the system comprising: a hardware processor that is programmed to: receive a file identified as corresponding to a first file type from a first source; generate a byte value statistical distribution of the data included in the file received from the first source; select a model byte value statistical distribution representative of the first file type from model byte value statistical distributions representative of a plurality of file types; determine a distance metric between the byte value statistical distribution of the data included in the file and the selected model byte value statistical distribution; and verify that a file type of the received file is the first file type based on a comparison of the distance metric to a distance metric threshold indicating that the file type of received file is the first file type. ### Claim 78 (depends on 77) The system of claim 77, wherein the hardware processor is further programmed to generate an alert that the file type of the received file is not the first file type based on the comparison of the distance metric to the distance metric threshold indicating that the received file is not of the first file type. ### Claim 79 (depends on 77) The system of claim 77, wherein the hardware processor is further configured to inhibit the file from being executed based on the comparison of the distance metric to the distance metric threshold indicating that the received file is not of the first file type. ### Claim 80 (depends on 77) The system of claim 77, wherein the hardware processor is further configured to cause the file to be tested to determine whether the file includes a malicious program based on the comparison of the distance metric to the distance metric threshold indicating that the received file is not of the first file type. ### Claim 81 (depends on 77) The system of claim 77, wherein the byte value statistical distribution of the file is a byte value distribution of the average frequency and variance of data contained in the file, and the selected model distribution is a byte value distribution of the average frequency and variance representative of files of the first file type. ### Claim 82 (depends on 77) The system of claim 77, wherein the byte value statistical distribution of the file is a byte value distribution of the data contained in the file, and the selected model distribution is a byte value distribution of representative of files of the first file type. ### Claim 83 (depends on 82) The system of claim 82, wherein the byte value distribution of the file is a byte frequency count of data contained in the file, and the selected model distribution is a byte frequency count for files of the first file type. ### Claim 84 (depends on 82) The system of claim 82, wherein the byte value distribution of the file is a rank ordered byte frequency count of data contained in the file, and the selected model distribution is a rank ordered byte frequency count for files of the first file type. ### Claim 85 (depends on 77) The system of claim 77, wherein the distance metric is calculated based on a Mahalanobis distance between the byte value statistical distribution of the data included in the file and the selected model byte value statistical distribution. ### Claim 86 A non-transitory computer-readable medium containing instructions that, when executed by a processor, cause the processor to perform a method for verifying a file type, the method comprising: receiving a file identified as corresponding to a first file type from a first source; generating a byte value statistical distribution of the data included in the file received from the first source; selecting a model byte value statistical distribution representative of the first file type from model byte value statistical distributions representative of a plurality of file types; determining a distance metric between the byte value statistical distribution of the data included in the file and the selected model byte value statistical distribution; and verifying that a file type of the received file is the first file type based on a comparison of the distance metric to a distance metric threshold indicating that the file type of received file is the first file type.