# US 2021/0240691 A1 · Anomaly identification in log files > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240691 A1 - Document ID: US20210240691A1 - Status: Published application - Kind: A1 - Application number: 16/776,891 - Filed: 2020-01-30 - Published: 2021-08-05 - Earliest claimed: 2020-01-30 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Tiago Bertoni Scarton; Dashiell Lucena Gonzaga Borges; Thiago Bianchi; Joao Souto Maior - Web page: https://patentgrep.com/patent/US20210240691A1 ## CPC - G06F 16/2365: {Ensuring data consistency and integrity} - G06F 11/0751: {Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06)} - G06F 11/302: {where the computing system component is a software system} - G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)} - G06F 17/40: Data acquisition and logging (for input to computer G06F3/00) - G06F 40/205: Parsing - G06F 40/242: Dictionaries - G06F 40/247: Thesauruses; Synonyms - G06F 40/30: Semantic analysis - G06Q 30/04: Billing or invoicing - G06F 2201/865: Monitoring of software - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract Described are techniques for identifying anomalies in log files. The techniques including a method comprising mapping a plurality of log entries into a bag-of-words matrix and determining respective cosine similarities between consecutive log entries in the bag-of-words matrix. The method further comprises converting respective cosine similarities to respective states and determining respective state transitions between consecutive log entries. The method further comprises generating a stochastic matrix based on the respective state transitions and identifying an anomalous log entry based on an anomalous state transition. The anomalous state transition has a probability below a probability threshold, and the probability is based on the stochastic matrix. ## Claims ### Claim 1 A computer-implemented method comprising: mapping a plurality of log entries into a bag-of-words matrix; determining respective cosine similarities between consecutive log entries in the bag-of-words matrix; converting the respective cosine similarities to respective states; determining respective state transitions between the consecutive log entries; generating a stochastic matrix based on the respective state transitions; and identifying an anomalous log entry based on an anomalous state transition that has a probability below a probability threshold, wherein the probability is based on the stochastic matrix. ### Claim 2 (depends on 1) The method of claim 1, wherein the plurality of log entries are derived from a log file, and wherein the method further comprises: segmenting, by a parser, the log file into the plurality of log entries, wherein the log file is segmented based on timestamp information associated with the plurality of log entries. ### Claim 3 (depends on 1) The method of claim 1, wherein mapping the plurality of log entries into the bag-of-words matrix further comprises: defining a dictionary of terms from the plurality of log entries; creating a first vector for a first log entry, the first vector comprising a respective count of each respective term in the first log entry; and adding the first vector to the bag-of-words matrix. ### Claim 4 (depends on 3) The method of claim 3, wherein defining the dictionary of terms further comprises: removing stop words from the plurality of log entries; converting inflected words to a corresponding etymological root form; and merging synonymous terms based on a semantic context of the plurality of log entries. ### Claim 5 (depends on 1) The method of claim 1, wherein identifying the anomalous log entry further comprises: calculating the probability of the anomalous state transition by dividing an element in a given row in the stochastic matrix corresponding to the anomalous state transition by a sum of elements in the given row; and determining that the probability is less than the probability threshold. ### Claim 6 (depends on 1) The method of claim 1, wherein the stochastic matrix comprises a respective number of occurrences for the respective state transitions. ### Claim 7 (depends on 1) The method of claim 1, wherein the method is performed by a host system according to software that is downloaded to the host system from a remote data processing system. ### Claim 8 (depends on 7) The method of claim 7, wherein the method further comprises: metering a usage of the software; and generating an invoice based on metering the usage. ### Claim 9 A system comprising: one or more processors; and one or more computer-readable storage media collectively storing program instructions which, when executed by the one or more processors, are configured to cause the one or more processors to perform a method comprising: mapping a plurality of log entries into a bag-of-words matrix; determining respective cosine similarities between consecutive log entries in the bag-of-words matrix; converting the respective cosine similarities to respective states; determining respective state transitions between the consecutive log entries; generating a stochastic matrix based on the respective state transitions; and identifying an anomalous log entry based on an anomalous state transition that has a probability below a probability threshold, wherein the probability is based on the stochastic matrix. ### Claim 10 (depends on 9) The system of claim 9, wherein the plurality of log entries are derived from a log file, and wherein the method further comprises: segmenting, by a parser, the log file into the plurality of log entries, wherein the log file is segmented based on timestamp information associated with the plurality of log entries. ### Claim 11 (depends on 9) The system of claim 9, wherein mapping the plurality of log entries into the bag-of-words matrix further comprises: defining a dictionary of terms from the plurality of log entries; creating a first vector for a first log entry, the first vector comprising a respective count of each respective term in the first log entry; and adding the first vector to the bag-of-words matrix. ### Claim 12 (depends on 11) The system of claim 11, wherein defining the dictionary of terms further comprises: removing stop words from the plurality of log entries; converting inflected words to a corresponding etymological root form; and merging synonymous terms based on a semantic context of the plurality of log entries. ### Claim 13 (depends on 9) The system of claim 9, wherein identifying the anomalous log entry further comprises: calculating the probability of the anomalous state transition by dividing an element in a given row in the stochastic matrix corresponding to the anomalous state transition by a sum of elements in the given row; and determining that the probability is less than the probability threshold. ### Claim 14 (depends on 9) The system of claim 9, wherein the stochastic matrix comprises a respective number of occurrences for the respective state transitions. ### Claim 15 A computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions comprising instructions configured to cause one or more processors to perform a method comprising: mapping a plurality of log entries into a bag-of-words matrix; determining respective cosine similarities between consecutive log entries in the bag-of-words matrix; converting the respective cosine similarities to respective states; determining respective state transitions between the consecutive log entries; generating a stochastic matrix based on the respective state transitions; and identifying an anomalous log entry based on an anomalous state transition that has a probability below a probability threshold, wherein the probability is based on the stochastic matrix. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the plurality of log entries are derived from a log file, and wherein the method further comprises: segmenting, by a parser, the log file into the plurality of log entries, wherein the log file is segmented based on timestamp information associated with the plurality of log entries. ### Claim 17 (depends on 15) The computer program product of claim 15, wherein mapping the plurality of log entries into the bag-of-words matrix further comprises: defining a dictionary of terms from the plurality of log entries; creating a first vector for a first log entry, the first vector comprising a respective count of each respective term in the first log entry; and adding the first vector to the bag-of-words matrix. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein defining the dictionary of terms further comprises: removing stop words from the plurality of log entries; converting inflected words to a corresponding etymological root form; and merging synonymous terms based on a semantic context of the plurality of log entries. ### Claim 19 (depends on 15) The computer program product of claim 15, wherein identifying the anomalous log entry further comprises: calculating the probability of the anomalous state transition by dividing an element in a given row in the stochastic matrix corresponding to the anomalous state transition by a sum of elements in the given row; and determining that the probability is less than the probability threshold. ### Claim 20 (depends on 15) The computer program product of claim 15, wherein the stochastic matrix comprises a respective number of occurrences for the respective state transitions.