# US 2022/0036002 A1 · Log sourcetype inference model training for a data intake and query system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0036002 A1 - Document ID: US20220036002A1 - Status: Published application - Kind: A1 - Application number: 16/945,448 - Filed: 2020-07-31 - Published: 2022-02-03 - Earliest claimed: 2020-07-31 - Assignees: Splunk Inc. - Applicants: Splunk Inc. - Inventors: Ram Sriharsha; Zhaohui Wang - Web page: https://patentgrep.com/patent/US20220036002A1 ## CPC - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 16/3347: {using vector based model} - G06F 16/355: Creation or modification of classes or clusters - G06F 40/242: Dictionaries - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods are described for training an artificial intelligence model to infer a log sourcetype of a log. For example, logs may have different log sourcetypes, and logs having the same log sourcetypes may have different messagetypes. The artificial intelligence model may be a machine learning model, and can be trained using training data that includes logs with known log sourcetypes. Each log can be tokenized, filtered, converted into a vector, and applied to a machine learning model as an input to perform the training. The machine learning model may output an inferred log sourcetype, which can be compared with the known log sourcetype to update model parameters to improve the machine learning model accuracy. The trained machine learning model may be trained to infer a log sourcetype of a log regardless of the messagetype of the log. ## Claims ### Claim 1 A computer-implemented method, comprising: obtaining a log generated by one or more components in an information technology environment; generating one or more tokens based on text in the log; filtering the one or more tokens based on a language dictionary to form a subset of one or more tokens; converting the subset of the one or more tokens into a vector; and training, using the vector, a machine learning model to predict a log sourcetype of a log applied to the trained machine learning model as an input. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein training the machine learning model further comprises: obtaining a second log; generating one or more second tokens based on text in the second log; filtering the one or more second tokens based on the language dictionary to form a second subset of one or more second tokens; converting the second subset of the one or more second tokens into a second vector; and training, using the vector and the second vector, the machine learning model. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein training the machine learning model further comprises: determining an inferred log sourcetype of the log based at least in part on applying the vector as an input to the machine learning model; and comparing the inferred log sourcetype with a known log sourcetype of the log. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein training the machine learning model further comprises: determining an inferred log sourcetype of the log based at least in part on applying the vector as an input to the machine learning model; comparing the inferred log sourcetype with a known log sourcetype of the log; and generating updated model parameters for the machine learning model in response to a determination that the inferred log sourcetype differs from the known log sourcetype by more than a previously determined loss. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein training the machine learning model further comprises: determining an inferred log sourcetype of the log based at least in part on applying the vector as an input to the machine learning model; comparing the inferred log sourcetype with a known log sourcetype of the log; and generating updated model parameters for the machine learning model based on a step size selected for a training algorithm in response to a determination that the inferred log sourcetype differs from the known log sourcetype by more than a previously determined loss. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein training the machine learning model further comprises: determining an inferred log sourcetype of the log based at least in part on applying the vector as an input to the machine learning model; comparing the inferred log sourcetype with a known log sourcetype of the log; and generating updated model parameters for the machine learning model based on a step size selected for a training algorithm and a value of a difference between the inferred log sourcetype and the known log sourcetype in response to a determination that the inferred log sourcetype differs from the known log sourcetype by more than a previously determined loss. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein training the machine learning model further comprises: determining an inferred log sourcetype of the log based at least in part on applying the vector as an input to the machine learning model; comparing the inferred log sourcetype with a known log sourcetype of the log; and refraining from generating updated model parameters for the machine learning model in response to a determination that the inferred log sourcetype differs from the known log sourcetype by less than a previously determined loss. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein training the machine learning model further comprises: determining an inferred log sourcetype of the log based at least in part on applying the vector as an input to the machine learning model; comparing the inferred log sourcetype with a known log sourcetype of the log; and outputting a difference between the inferred log sourcetype and the known log sourcetype to a component in a data processing pipeline. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein training the machine learning model further comprises: determining an inferred log sourcetype of the log based at least in part on applying the vector as an input to the machine learning model; and outputting the inferred log sourcetype to a component in a data processing pipeline. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, further comprising storing the trained machine learning model in a data store accessible by at least one of a data processing pipeline or a query system. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the language dictionary comprises an English language dictionary. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein filtering the one or more tokens based on a language dictionary further comprises removing at least one token from the one or more tokens that corresponds with a word not found in the language dictionary. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein training the machine learning model further comprises training the machine learning model using the vector and an online stochastic gradient descent (SGD) algorithm. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, wherein training the machine learning model further comprises training the machine learning model using the vector and a stochastic gradient descent (SGD) algorithm. ### Claim 15 (depends on 1) The computer-implemented method of claim 1, wherein converting the filtered one or more tokens into a vector further comprises generating the vector using the filtered one or more tokens and a hashing TF algorithm. ### Claim 16 (depends on 1) The computer-implemented method of claim 1, wherein converting the filtered one or more tokens into a vector further comprises generating the vector using the filtered one or more tokens and a CountVectorizer algorithm. ### Claim 17 A system comprising: a data store including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to: obtain a log generated by one or more components in an information technology environment; generate one or more tokens based on text in the log; filter the one or more tokens based on a language dictionary to form a subset of one or more tokens; convert the subset of the one or more tokens into a vector; and train, using the vector, a machine learning model to predict a log sourcetype of a log applied to the trained machine learning model as an input. ### Claim 18 (depends on 17) The system of claim 17, wherein execution of the computer-executable instructions further causes the system to: determine an inferred log sourcetype of the log based at least in part on applying the vector as an input to the machine learning model; compare the inferred log sourcetype with a known log sourcetype of the log; and generate updated model parameters for the machine learning model in response to a determination that the inferred log sourcetype differs from the known log sourcetype by more than a previously determined loss. ### Claim 19 Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to: obtain a log generated by one or more components in an information technology environment; generate one or more tokens based on text in the log; filter the one or more tokens based on a language dictionary to form a subset of one or more tokens; convert the subset of the one or more tokens into a vector; and train, using the vector, a machine learning model to predict a log sourcetype of a log applied to the trained machine learning model as an input. ### Claim 20 (depends on 19) The non-transitory computer-readable media of claim 19, wherein the computer-executable instructions, when executed, further cause the computing system to: determine an inferred log sourcetype of the log based at least in part on applying the vector as an input to the machine learning model; compare the inferred log sourcetype with a known log sourcetype of the log; and generate updated model parameters for the machine learning model in response to a determination that the inferred log sourcetype differs from the known log sourcetype by more than a previously determined loss.