Block anchors for online log parsing
AssigneeAmazon Technologies, Inc.
InventorsXiao Wang, Jasmeet Chhabra
Life of this patent
Estimating the term…
Abstract
Online block parsing techniques are performed that use anchor tokens. When a log entry of a log stream is received, the log entry may be tokenized into an ordered series of log units. The log units may be matched with anchor tokens in previously initialized log templates to determine a matching log template. A log template based on the matching log template may be provided with the tokenized log entry to a destination for a parsed version of the log stream.
Claims21
-20. (canceled)
A system, comprising:
- one or more processors;
- and
- a memory, that stores program instructions that, when executed by the at least one processor, cause the one or more processors to implement an online log parser, configured to:
- determine that a log template matches a log entry obtained from a log stream, based, at least in part, on one or more anchor tokens for a previous received log entry that matches one of the one or more log token units in a ordered series of one or more log token units of the log entry;
- determine one or more updates to the log template;
- apply the one or more updates to update the log template;
- determine that the updated log template matches a further log entry obtained from the log stream;
- and
- provide the ordered series of the one or more log token units and a log template based on the previously initialized log template to a destination for a parsed version of the log stream.
A method, comprising:
- determining, by an online log parser, that a log template matches a log entry obtained from a log stream, based, at least in part, on one or more anchor tokens for a previous received log entry that matches one of the one or more log token units in a ordered series of one or more log token units of the log entry;
- determining, by the online log parser, one or more updates to the log template;
- applying, by the online log parser, the one or more updates to update the log template;
- determining, by the online log parser, that the updated log template matches a further log entry obtained from the log stream;
- and
- providing, by the online log parser, the ordered series of the one or more log token units and a log template based on the previously initialized log template to a destination for a parsed version of the log stream.
One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement:
- determining, by an online log parser, that a log template matches a log entry obtained from a log stream, based, at least in part, on one or more anchor tokens for a previous received log entry that matches one of the one or more log token units in a ordered series of one or more log token units of the log entry;
- determining, by the online log parser, one or more updates to the log template;
- applying, by the online log parser, the one or more updates to update the log template;
- determining, by the online log parser, that the updated log template matches a further log entry obtained from the log stream;
- and
- providing, by the online log parser, the ordered series of the one or more log token units and a log template based on the previously initialized log template to a destination for a parsed version of the log stream.
Dependencies are inferred from the claim text.