System and method for optimally managing heterogeneous data in a distributed storage environment

A data management computing apparatus comprising:

• a processor;
• and
• a memory coupled to the processor, which is configured for executing programmed instructions comprising and stored in the memory to:
• initialize a first storage pool capable of storing data from one or more sources, the first storage pool being distributed across one or more computing devices;
• store data from the one or more sources in the first storage pool;
• generate one or more memory pools in a second storage pool based on amount of data to be stored in the first storage pool and one or more parameters associated with the data stored in the first storage pool, the second storage pool being distributed across the one or more computing devices;
• create metadata in a first memory pool of the one or more memory pools for the data stored in the first storage pool, the metadata capable of retrieving the data stored in the first storage pool in real-time;
• identify a failure in at least one of the one or more computing devices;
• circulate the information regarding the failure to the metadata stored in the first storage pool;
• create one or more redundant copies of the data corrupted by the failure;
• and
• update the metadata in response to creating one or more redundant copies of the data.

A non-transitory computer readable medium having stored thereon instructions for optimally managing data in a distributed storage environment comprising executable code which when executed by a processor, causes the processor to perform steps comprising:

• initializing a first storage pool capable of storing data from one or more sources, the first storage pool being distributed across one or more computing devices;
• storing data from one or more sources in the first storage pool;
• generating one or more memory pools in a second storage pool based on amount of data to be stored in the first storage pool and one or more parameters associated with the data stored in the first storage pool, the second storage pool being distributed across the one or more computing devices;
• creating metadata in a first memory pool of the one or more memory pools for the data stored in the first storage pool, the metadata capable of retrieving the data stored in the first storage pool in real-time;
• identifying a failure in at least one of the one or more computing devices;
• circulating the information regarding the failure to the metadata stored in the first storage pool;
• creating one or more redundant copies of the data corrupted by the failure;
• and
• updating the metadata in response to creating one or more redundant copies of the data.