Method and system for data archiving and retrieval in distributed search and analytics environment

A computing device configured to implement an execution of a method for data archiving in a distributed search and analytics environment, the computing device comprising:

• a processor;
• a memory;
• and
• a communication interface coupled to each of the processor and the memory, wherein the processor is configured to:
• ingest a real-time data stream from at least one data source;
• segment the ingested data stream into at least one segment and storing each of the at least one segment across a set of nodes in at least one search engine cluster;
• capture a set of resiliency snapshots of the at least one segment and storing the captured set of resiliency snapshots of the at least one segment within the at least one search engine cluster after a first predefined time period;
• create at least one index snapshot based on the stored set of resiliency snapshots through a clone Application Programming Interface (API) after a second predefined time period, wherein the at least one index snapshot is a reference to the stored set of resiliency snapshots, wherein the set of resiliency snapshots captures a current state of the at least one index snapshot, and wherein the current state represents a specific configuration, data, mappings, and settings that exist in the at least one index snapshot at the time the set of resiliency snapshots is captured;
• mount the at least one index snapshot onto an archival search engine cluster using a mount option, wherein the archival search engine cluster comprises backup nodes associated with a storage system;
• and
• delete the at least one index snapshot from the at least one search engine cluster after mounting onto the archival search engine cluster, wherein data associated with the at least one index snapshot is migrated from active or archival data nodes to the backup nodes, thereby reducing requirement of the shard size and overall volume that needs to be managed by the at least one search engine cluster.

A non-transitory computer-readable storage medium storing instruction for data archiving in a distributed search and analytics environment, the storage medium comprising executable code which, when executed by a processor, caused the processor to:

• ingest a real-time data stream from at least one data source;
• segment the ingested data stream into at least one segment and storing each of the at least one segment across a set of nodes in at least one search engine cluster;
• capture a set of resiliency snapshots of the at least one segment and storing the captured set of resiliency snapshots of the at least one segment within the at least one search engine cluster after a first predefined time period;
• create at least one index snapshot based on the stored set of resiliency snapshots through a clone Application Programming Interface (API) after a second predefined time period, wherein the at least one index snapshot is a reference to the stored set of resiliency snapshots, wherein the set of resiliency snapshots captures a current state of the at least one index snapshot, and wherein the current state represents a specific configuration, data, mappings, and settings that exist in the at least one index snapshot at the time the set of resiliency snapshots is captured;
• mount the at least one index snapshot onto an archival search engine cluster using a mount option, wherein the archival search engine cluster comprises backup nodes associated with a storage system;
• and
• delete the at least one index snapshot from the at least one search engine cluster after mounting onto the archival search engine cluster, wherein data associated with the at least one index snapshot is migrated from active or archival data nodes to the backup nodes, thereby reducing requirement of the shard size and overall volume that needs to be managed by the at least one search engine cluster.