Tiered heterogeneous fast layer shared storage substrate apparatuses, methods, and systems

A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for serving data in a distributed network, the operations comprising:

• associating metadata and data to any of a plurality of distributed nodes of a storage system such that a global copy of the metadata does not exist in one or more fixed locations known to all distributed nodes by at least assigning each request to write new data to the storage system to a local node of the plurality of distributed nodes, wherein the storage system comprises a plurality of block devices configured to store data, and wherein the plurality of block devices are communicatively coupled to different ones of the plurality of distributed nodes;
• maintain, by each respective node of the plurality of distributed nodes, precise metadata for data stored at one or more block devices of the plurality of block devices local to the respective node without maintaining any metadata for data stored at other block devices of the plurality of block devices local to other nodes of the plurality of distributed nodes receiving data request at a distributed node of the plurality of distributed nodes from a locally running application via a network;
• determining, by the distributed node, whether the data request is a local data request based on the metadata maintained by the distributed node, wherein a local data request corresponds to a data request associated with data stored at one or more block devices local to the distributed node and a remote data request corresponds to a data request associated with data stored at other block devices local to other distributed nodes of the plurality of distributed nodes;
• when the data request is the local data request, using metadata maintained by the distributed node to serve data corresponding to the data request;
• when the data is the remote data request, querying the other distributed nodes to identify one or more of the other distributed nodes local to at least one block device storing the data associated with the data request serving the requested data based on the data stored local to a particular one of the identified other nodes;
• determining, by the distributed node, performance metrics including at least one of a latency associated with data served from the one or more block devices local to the distributed node, energy efficiency information associated with serving requested data from the one or more block devices local to the distributed node, and information indicating a frequency of access with respect to the data stored at the one or more block devices local to the distributed node;
• and
• migrating, by the distributed node, particular data stored at the one or more block devices local to the distributed node to one of the other block devices local to the other nodes based on a determination that migration of the particular data improves at least one of the performance metrics.

A processor-implemented method of organizing and maintaining metadata and serving data in a distributed network, the method comprising:

• associating, by a management server, metadata to any of a plurality of distributed nodes on a network, wherein each of the plurality of distributed nodes is communicatively coupled to a block device of a plurality of block devices, such that a global copy of the metadata does not exist in one or more fixed locations known to all distributed nodes, and wherein the associating the metadata and the data to different nodes of the plurality of nodes comprises:
• assigning each request to write new data to a storage system to a local node of the plurality of distributed nodes;
• maintaining, by each respective node of the plurality of distributed nodes, precise metadata for data stored at the block device associated with the respective node without maintaining any metadata for data stored at other block devices of the plurality of block devices associated with other nodes of the plurality of distributed nodes;
• determining, by a particular node of the plurality of distributed nodes, whether a received data request is a local data request or a remote data request based on particular metadata maintained by the particular node, wherein a local data request corresponds to a data request associated with data stored at the block device associated with the particular node and a remote data request corresponds to a data request associated with data stored at other block devices associated with the other nodes;
• serving, by the particular node, requested data associated with the received data request based on the data stored at the block device associated with the particular node in response to a local data request;
• in response to a remote data request:
• querying the other nodes to identify one or more of the other nodes associated with at least one block device storing the data associated with the data request;
• and
• serving requested data associated with the received data request based on the data stored at one of the at least one block devices associated with one of the identified other nodes;
• determining, by the particular node, performance metrics including at least one of a latency associated with data served from the one or more block devices local to the respective node, energy efficiency information associated with serving requested data from the one or more block devices local to the respective node, and information indicating a frequency of access with respect to the data stored at the one or more block devices local to the respective node;
• and
• migrating, by the particular node, particular data stored at the block device associated with the particular node to one of the other block devices associated with the other nodes based on a determination that migration of the particular data improves at least one of the performance metrics.