Systems and methods for optimizing selection of a replication data node in a distributed file system
AssigneeDell Products L.P.
InventorsKesava Vijaya Krupakaran, Balaji Venkat Venkataswami
Life of this patent
Estimating the term…
Abstract
Aspects of the present invention include optimizing selection of a replication data node in a Hadoop Distributed File System. In embodiments of the present invention, optimizing the replication data node selection can be based on a priorities based algorithm. In embodiments of the present invention, optimizing replication data node selection can be based on a relative comparison of parameters.
Claims20
A storage system for managing replicas of data blocks distributed across a storage cluster, the system comprising:
- one or more processors comprising memory;
- a Software Defined Network (SDN) controller communicatively coupled to the one or more processors to receive network data, the SDN controller listens to link state advertisements to generate a database comprising link states and uses the link states and one or more network parameters to determine a relative distance measure between a source node comprising a source block and candidate replication data nodes, the relative distance measure being obtained using one or more user-configurable weights;
- and
- a name node that is different from the source node and is communicatively coupled to the SDN controller, the name node being a centralized node that ascertains where data blocks are stored, queries the SDN controller to obtain at least the relative distance measure, and uses the relative distance measure to select from the candidate replication data nodes a set of data nodes at which to replicate the source block.
A method for managing replicas of a data blocks distributed across a storage cluster in a network, the system comprising:
- receiving, from a Software Defined Network (SDN) controller, network data comprising network parameters, the SDN controller listens to link state advertisements to generate a database that comprises link states;
- based on at least some of the link state and the network data, determining a relative distance measure between a source node and candidate replication data nodes,
- the source node comprising a source block, the relative distance measure being obtained using one or more user-configurable weights;
- in response to being queried by a name node that is different from the source node, providing the relative distance measure to the name node to select from the candidate replication data nodes a set of data nodes at which to replicate the source block, the name node being a centralized node that ascertains where data blocks are stored;
- replicating the source block to obtain a replicated source block;
- and
- storing the replicated source block in the set of data nodes.
A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes steps for managing replicas of data blocks distributed across a storage cluster comprising:
- receiving, from a Software Defined Network (SDN) controller, network data comprising network parameters, the SDN controller listens to link state advertisements to generate a database that comprises link states;
- based on at least some of the network data and the link states, determining a relative distance measure between a source node and candidate replication data nodes, the source node comprising a source block, the relative distance measure being obtained using one or more user-configurable weights;
- and
- in response to being queried by a name node providing the relative distance measure to the name node, the name node being a centralized node that ascertains where data blocks are stored and using the relative distance measure to select from the source node that is different from the name node a set of data nodes at which to replicate the source block.
Dependencies are inferred from the claim text.