# US 2025/0094253 A1 · Indexing data at a data intake and query system based on a node capacity threshold > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0094253 A1 - Document ID: US20250094253A1 - Status: Published application - Kind: A1 - Application number: 18/641,089 - Filed: 2024-04-19 - Published: 2025-03-20 - Earliest claimed: 2020-07-31 - Assignees: Splunk Inc. - Applicants: Splunk Inc. - Inventors: Shalabh Goyal; Anish Shrigondekar; Bhavin Thaker; Zhenghui Xie; Ruochen Zhang - Web page: https://patentgrep.com/patent/US20250094253A1 ## CPC - G06F 9/546: {Message passing systems or structures, e.g. queues} - G06F 11/3006: {where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50)} - G06F 11/3409: {for performance assessment} - G06F 2201/81: Threshold - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract As an indexer indexes and groups events, it can generate data slices that include events. Based on a slice rollover policy, the indexer can add a particular slice to an aggregate slice. Based on an aggregate slice backup policy, the indexer can store a copy of the aggregate slice to a shared storage system. The aggregate slice can be used for restore purposes in the event the indexer fails or becomes unresponsive. ## Claims ### Claim 1 (canceled) ### Claim 2 A method, comprising: receiving, at an indexing node of a data intake and query system, a message payload including a first plurality of events; extracting the first plurality of events from the message payload; adding two or more events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, the hot bucket including at least one index file, and a plurality of aggregate data slices; based on a slice rollover policy, converting the first editable data slice to a first non-editable data slice and adding the first non-editable data slice to a first aggregate data slice of the plurality of aggregate data slices, wherein the first aggregate data slice comprises a plurality of non-editable data slices, each of the plurality of non-editable data slices comprising a second plurality of events; concurrent to at least one of adding the two or more events to the first editable data slice or adding the first non-editable data slice to the first aggregate data slice, modifying one or more files of the hot bucket using the two or more events; based on an aggregate slice backup policy and a determination that a warm bucket corresponding to the hot bucket has not been stored on a remote shared storage system, communicating a copy of the first aggregate data slice to the remote shared storage system for storage, wherein the copy of the first aggregate data slice is communicated independent of the hot bucket; and based on a bucket rollover policy, converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice. ### Claim 3 (depends on 2) The method of claim 2, wherein each event of the first plurality of events includes raw machine data associated with a timestamp. ### Claim 4 (depends on 2) The method of claim 2, wherein extracting the first plurality of events comprises decoding the first plurality of events. ### Claim 5 (depends on 2) The method of claim 2, wherein the two or more events of the first plurality of events are added to the first editable data slice based on an association between the first plurality of events with a particular index, wherein the particular index is associated with the first editable data slice and the hot bucket. ### Claim 6 (depends on 2) The method of claim 2, wherein adding the first non-editable data slice to the first aggregate data slice includes compressing the first non-editable data slice, and wherein the first non-editable data slice is added to the first aggregate data slice as a compressed first non-editable data slice. ### Claim 7 (depends on 2) The method of claim 2, wherein the slice rollover policy includes a slice size threshold, and wherein the first non-editable data slice is added to the first aggregate data slice based on a determination that a size of the first non-editable data slice satisfies the slice size threshold. ### Claim 8 (depends on 2) The method of claim 2, wherein the slice rollover policy includes a slice time threshold, and wherein the first non-editable data slice is added to the first aggregate data slice based on a determination that an amount of time since the first non-editable data slice was generated satisfies the slice time threshold. ### Claim 9 (depends on 2) The method of claim 2, wherein the aggregate slice backup policy includes a slice size threshold, and wherein the copy of the first aggregate data slice is stored to the remote shared storage system based on a determination that a size of the first aggregate data slice satisfies the slice size threshold. ### Claim 10 (depends on 2) The method of claim 2, wherein the aggregate slice backup policy includes a slice time threshold, and wherein the copy of the first aggregate data slice is stored to the remote shared storage system based on a determination that an amount of time since the first aggregate data slice was generated satisfies the slice time threshold. ### Claim 11 (depends on 2) The method of claim 2, further comprising based on the bucket rollover policy, generating a second hot bucket. ### Claim 12 (depends on 2) The method of claim 2, wherein the bucket rollover policy includes a bucket size threshold, wherein converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage is further based on a determination that a size of the hot bucket satisfies the bucket size threshold. ### Claim 13 (depends on 2) The method of claim 2, wherein the bucket rollover policy includes a bucket timing threshold, wherein converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage is further based on a determination that an amount of time since the hot bucket was generated satisfies the bucket timing threshold. ### Claim 14 (depends on 2) The method of claim 2, further comprising, based on a determination that the copy of the first warm bucket is stored on the remote shared storage system, removing the copy of the first aggregate data slice from the remote shared storage system. ### Claim 15 (depends on 2) The method of claim 2, wherein the message payload is received from a remote message bus, and wherein the method further comprises: identifying for each event of the first plurality of events, a particular bucket that includes the event; and based on a determination that a copy of each bucket that includes at least one event of the first plurality of events was stored to the remote shared storage system, communicating an acknowledgement to the remote message bus, wherein the remote message bus deletes the message payload in response to the acknowledgement. ### Claim 16 (depends on 2) The method of claim 2, further comprising, based at least in part on communicating the copy of the first aggregate data slice, generating an additional aggregate data slice, wherein the additional aggregate data slice is associated with the hot bucket. ### Claim 17 A computing device, comprising: memory; and one or more processing devices communicatively coupled to the memory and configured to: receive, at an indexing node of a data intake and query system, a message payload including a first plurality of events; extract the first plurality of events from the message payload; add two or more events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, the hot bucket including at least one index file, and a plurality of aggregate data slices; based on a slice rollover policy, convert the first editable data slice to a first non-editable data slice and adding the first non-editable data slice to a first aggregate data slice of the plurality of aggregate data slices, wherein the first aggregate data slice comprises a plurality of non-editable data slices, each of the plurality of non-editable data slices comprising a second plurality of events; concurrent to at least one of addition of the two or more events to the first editable data slice or addition of the first non-editable data slice to the first aggregate data slice, modify one or more files of the hot bucket using the two or more events; based on an aggregate slice backup policy and a determination that a warm bucket corresponding to the hot bucket has not been stored on a remote shared storage system, communicate a copy of the first aggregate data slice to the remote shared storage system for storage, wherein the copy of the first aggregate data slice is communicated independent of the hot bucket; and based on a bucket rollover policy, convert the hot bucket to a first warm bucket and communicate a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice. ### Claim 18 (depends on 17) The computing device of claim 17, wherein each event of the first plurality of events includes raw machine data associated with a timestamp. ### Claim 19 (depends on 17) The computing device of claim 17, wherein the two or more events of the first plurality of events are added to the first editable data slice based on an association between the first plurality of events with a particular index, wherein the particular index is associated with the first editable data slice and the hot bucket. ### Claim 20 (depends on 17) The computing device of claim 17, wherein the slice rollover policy includes a slice size threshold, and wherein the first non-editable data slice is added to the first aggregate data slice based on a determination that a size of the first non-editable data slice satisfies the slice size threshold. ### Claim 21 Non-transitory computer-readable media comprising computer-executable instructions that when executed by one or more processing devices, causes the one or more processing devices to: receive, at an indexing node of a data intake and query system, a message payload including a first plurality of events; extract the first plurality of events from the message payload; add two or more events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, the hot bucket including at least one index file, and a plurality of aggregate data slices; based on a slice rollover policy, convert the first editable data slice to a first non-editable data slice and adding the first non-editable data slice to a first aggregate data slice of the plurality of aggregate data slices, wherein the first aggregate data slice comprises a plurality of non-editable data slices, each of the plurality of non-editable data slices comprising a second plurality of events; concurrent to at least one of addition of the two or more events to the first editable data slice or addition of the first non-editable data slice to the first aggregate data slice, modify one or more files of the hot bucket using the two or more events; based on an aggregate slice backup policy and a determination that a warm bucket corresponding to the hot bucket has not been stored on a remote shared storage system, communicate a copy of the first aggregate data slice to the remote shared storage system for storage, wherein the copy of the first aggregate data slice is communicated independent of the hot bucket; and based on a bucket rollover policy, convert the hot bucket to a first warm bucket and communicate a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice.