Back to results
US 11,294,941 B1Granted
Message-based data ingestion to a data intake and query system
AssigneeSplunk Inc.
InventorsEric Sammer, Sourav Pal, Joseph Gabriel Echeverria
Life of this patent
Estimating the term…
Abstract
Systems and methods are described for preprocessing data later ingested into an indexing system. The preprocessing can include receiving messages published to a first publish-subscribe messaging system, the messages containing raw machine data generated by one or more components in an information technology environment, performing one or more processing operations on at least some of the messages to generate preprocessed messages, republishing the preprocessed messages to a second publish-subscribe messaging system, and providing to the indexing system, a subset of the messages from the second publish-subscribe messaging system.
Claims29
A method, comprising:
- receiving messages published to a first publish-subscribe messaging system, the messages containing raw machine data generated by one or more components in an information technology environment;
- at the first publish-subscribe messaging system:
- performing one or more processing operations on at least some of the messages to generate preprocessed messages;
- republishing the preprocessed messages from the first publish-subscribe messaging system to a second publish-subscribe messaging system;
- resiliently storing the preprocessed messages at the first publish-subscribe messaging system at least until confirmation is received from the second publish-subscribe messaging system that the preprocessed messages have been republished at the second publish-subscribe messa in system;
- and
- at the second publish-subscribe messaging system:
- transmitting confirmation to the first publish-subscribe messaging system that the preprocessed messages have been republished at the second publish-subscribe messaging system only after the republishing of the preprocessed messages to the second publish-subscribe messaging system completes;
- and
- providing to an indexing system, a subset of the messages from the second publish-subscribe messaging system;
- wherein the method further comprising deleting the preprocessed messages at the first publish-subscribe messaging system subseQuent receiving confirmation from the second publish-subscribe messaging system that the preprocessed messages have been republished at the second publish-subscribe messaging system.
8 more elements
A system comprising:
- a data store including computer-executable instructions;
- and
- one or more processors configured to execute the computer-executable instructions to implement a first publish-subscribe messaging system and a second publish-subscribe messaging system, wherein execution of the computer-executable instructions causes the system to:
- at the first publish-subscribe messaging system:
- receive messages published to a first publish-subscribe messaging system, the messages containing raw machine data generated by one or more components in an information technology environment;
- perform one or more processing operations on at least some of the messages to generate preprocessed messages;
- republish the preprocessed messages from the first publish-subscribe messaging system to a second publish-subscribe messaging system;
- resiliently store the preprocessed messages at the first publish-subscribe messaging system at least until confirmation is received from the second publish-subscribe messaging system that the preprocessed messages have been republished at the second publish-subscribe messaging system;
- and
- at the second publish-subscribe messaging system:
- transmit confirmation to the first publish-subscribe messaging system that the preprocessed messages have been republished at the second publish-subscribe messaging system only after the republishing of the preprocessed messages to the second publish-subscribe messaging system completes;
- and
- provide to an indexing system, a subset of the messages from the second publish-subscribe messaging system;
- wherein execution of the computer-executable instructions causes the system to delete the preprocessed messages at the first publish-subscribe messaging system subsequent receiving confirmation from the second publish-subscribe messaging system that the preprocessed messages have been republished at the second publish-subscribe messaging system.
11 more elements
Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system implementing a first publish-subscribe messaging system and a second publish-subscribe messaging system, cause the computing system to:
- at the first publish-subscribe messaging system:
- receive messages published to a first publish-subscribe messaging system, the messages containing raw machine data generated by one or more components in an information technology environment;
- perform one or more processing operations on at least some of the messages to generate preprocessed messages;
- republish the preprocessed messages from the first publish-subscribe messaging system to a second publish-subscribe messaging system;
- resiliently store the preprocessed messages at the first publish-subscribe messaging system at least until confirmation is received from the second publish-subscribe messaging system that the preprocessed messages have been republished at the second publish-subscribe messaging system;
- and
- at the second publish-subscribe messaging system:
- transmit confirmation to the first publish-subscribe messaging system that the preprocessed messages have been republished at the second publish-subscribe messaging system only after the republishing of the preprocessed messages to the second publish-subscribe messaging system completes;
- and
- provide to an indexing system, a subset of the messages from the second publish-subscribe messaging system;
- wherein execution of the computer-executable instructions further causes the computing system to delete the preprocessed messages at the first publish-subscribe messaging system subsequent receiving confirmation from the second publish-subscribe messaging system that the preprocessed messages have been republished at the second publish-subscribe messaging system.
8 more elements
Dependencies are inferred from the claim text.