Indexing data at a data intake and query system based on a node capacity threshold

A method, comprising:

• receiving, at an indexing node of a data intake and query system, a message payload including a first plurality of events;
• extracting the first plurality of events from the message payload;
• adding two or more events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, the hot bucket including at least one index file, and a plurality of aggregate data slices;
• based on a slice rollover policy, converting the first editable data slice to a first non-editable data slice and adding the first non-editable data slice to a first aggregate data slice of the plurality of aggregate data slices, wherein the first aggregate data slice comprises a plurality of non-editable data slices, each of the plurality of non-editable data slices comprising a second plurality of events;
• concurrent to at least one of adding the two or more events to the first editable data slice or adding the first non-editable data slice to the first aggregate data slice, modifying one or more files of the hot bucket using the two or more events;
• based on an aggregate slice backup policy and a determination that a warm bucket corresponding to the hot bucket has not been stored on a remote shared storage system, communicating a copy of the first aggregate data slice to the remote shared storage system for storage, wherein the copy of the first aggregate data slice is communicated independent of the hot bucket;
• and
• based on a bucket rollover policy, converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice.

A computing device, comprising:

• memory;
• and
• one or more processing devices communicatively coupled to the memory and configured to:
• receive, at an indexing node of a data intake and query system, a message payload including a first plurality of events;
• extract the first plurality of events from the message payload;
• add two or more events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, the hot bucket including at least one index file, and a plurality of aggregate data slices;
• based on a slice rollover policy, convert the first editable data slice to a first non-editable data slice and adding the first non-editable data slice to a first aggregate data slice of the plurality of aggregate data slices, wherein the first aggregate data slice comprises a plurality of non-editable data slices, each of the plurality of non-editable data slices comprising a second plurality of events;
• concurrent to at least one of addition of the two or more events to the first editable data slice or addition of the first non-editable data slice to the first aggregate data slice, modify one or more files of the hot bucket using the two or more events;
• based on an aggregate slice backup policy and a determination that a warm bucket corresponding to the hot bucket has not been stored on a remote shared storage system, communicate a copy of the first aggregate data slice to the remote shared storage system for storage, wherein the copy of the first aggregate data slice is communicated independent of the hot bucket;
• and
• based on a bucket rollover policy, convert the hot bucket to a first warm bucket and communicate a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice.

Non-transitory computer-readable media comprising computer-executable instructions that when executed by one or more processing devices, causes the one or more processing devices to:

• receive, at an indexing node of a data intake and query system, a message payload including a first plurality of events;
• extract the first plurality of events from the message payload;
• add two or more events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, the hot bucket including at least one index file, and a plurality of aggregate data slices;
• based on a slice rollover policy, convert the first editable data slice to a first non-editable data slice and adding the first non-editable data slice to a first aggregate data slice of the plurality of aggregate data slices, wherein the first aggregate data slice comprises a plurality of non-editable data slices, each of the plurality of non-editable data slices comprising a second plurality of events;
• concurrent to at least one of addition of the two or more events to the first editable data slice or addition of the first non-editable data slice to the first aggregate data slice, modify one or more files of the hot bucket using the two or more events;
• based on an aggregate slice backup policy and a determination that a warm bucket corresponding to the hot bucket has not been stored on a remote shared storage system, communicate a copy of the first aggregate data slice to the remote shared storage system for storage, wherein the copy of the first aggregate data slice is communicated independent of the hot bucket;
• and
• based on a bucket rollover policy, convert the hot bucket to a first warm bucket and communicate a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice.