Dynamically converting search-time fields to ingest-time fields
AssigneeVMware, Inc.
InventorsJay A. Patel, Zhenmin Li, Chengdu Huang
Life of this patent
Estimating the term…
Abstract
Large amounts of unstructured or semi-structured log data generated by software and infrastructure components of a computing system are processed to identify anomalies and potential problems within the computing system. Stored log messages may be queried and analyzed according to dynamic fields constructed from the content of the log messages. As time goes on, the dynamic fields may be converted into static fields which are extracted and indexed at the time of ingestion of the log messages.
Claims20
A method for processing semi-structured data comprising:
- storing a first plurality of log messages in a first data store during a first time period;
- responsive to receiving a first query having a field, extracting first field values for the field from the first plurality of log messages at the time of issuance of the first query;
- storing a second plurality of log messages in a second data store during a second time period subsequent to the first time period;
- updating an index of the second data store for second field values for the field extracted from the second plurality of log messages at the time of storing the second plurality of log messages;
- and
- responsive to receiving a second query having the field during the second time period, retrieving the second field values for the field from the index of the second data store.
A non-transitory computer-readable storage medium comprising instructions that, when executed in a computing device, process semi-structured data, by performing the steps of:
- storing a first plurality of log messages in a first data store during a first time period;
- responsive to receiving a first query having a field, extracting first field values for the field from the first plurality of log messages at the time of issuance of the first query;
- storing a second plurality of log messages in a second data store during a second time period subsequent to the first time period;
- updating an index of the second data store for second field values for the field extracted from the second plurality of log messages at the time of storing the second plurality of log messages;
- and
- responsive to receiving a second query having the field during the second time period, retrieving the second field values for the field from the index of the second data store.
A computer system for processing semi-structured data for a computer infrastructure, the computer system comprising:
- a system memory;
- a storage device having a plurality of data stores and associated indexes;
- and
- a processor programmed to carry out the steps of:
- storing a first plurality of log messages in a first data store of the plurality of data stores during a first time period;
- responsive to receiving a first query having a field, extracting first field values for the field from the first plurality of log messages at the time of issuance of the first query;
- storing a second plurality of log messages in a second data store of the plurality of data stores during a second time period subsequent to the first time period;
- updating an index of the second data store for second field values for the field extracted from the second plurality of log messages at the time of storing the second plurality of log messages;
- and
- responsive to receiving a second query having the field during the second time period, retrieving the second field values for the field from the index of the second data store.
Dependencies are inferred from the claim text.