Data node fencing in a distributed file system
AssigneeCloudera, Inc.
InventorsTodd Lipcon, Aaron T. Myers, Eli Collins
Life of this patent
Estimating the term…
Abstract
Systems and methods for data node fencing in a distributed file system to prevent data inconsistencies and corruptions are disclosed. An embodiment includes implementing a protocol whereby data nodes detect a failover and determine an active name node based on transaction identifiers associated with transaction requests. The data nodes also provide to the active name node block location information and an acknowledgment. The embodiment further includes a protocol whereby a name node refrains from issuing invalidation requests to the data nodes until the name node receives acknowledgments from all data nodes that are functional.
Claims9
A method for maintaining data correctness in a Hadoop™ based distributed cluster during a failover, in which an original name node is switched to a backup name node due to failure of the original name node, the distributed cluster having a plurality of data nodes and one or more processors, the method being performed by the one or more processors and comprising:
- on the backup name node:
- assuming an active role to become a new active name node, upon detecting that the original name node has failed;
- flagging all of the plurality of data nodes as untrusted;
- for each data node among the plurality of data nodes:
- queuing, instead of issuing, commands intended for a data node until the data node is flagged as trusted, and
- upon receiving an acknowledgement from the data node acknowledging the assumption of the active role of the backup name node, flagging the data node as trusted;
- and
- on a respective data node:
- receiving a first command with a first transaction number from a first name node;
- receiving a second command with a second transaction number from a second name node, wherein the second transaction number is greater than the first transaction number;
- and
- sending an acknowledgment of an active role to the second name node.
A Hadoop™ based distributed cluster comprising an original name node, a backup name node, and a distributed file system having a plurality of data nodes,
- wherein one or more processors of the backup name node are configured to perform:
- assuming an active role to become a new active name node, upon detecting that the original name node has failed;
- flagging all of the plurality of data nodes as untrusted;
- for each data node among the plurality of data nodes:
- queuing, instead of issuing, commands intended for a data node until the data node is flagged as trusted;
- and
- upon receiving an acknowledgement from the data node acknowledging the assumption of the active role of the backup name node, flagging the data node as trusted, and
- wherein one or more processors of a respective data node are configured to perform;
- receiving a first command with a first transaction number from a first name node;
- receiving a second command with a second transaction number from a second name node, wherein the second transaction number is greater than the first transaction number;
- and
- sending an acknowledgment of an active role to the second name node.
A machine-readable storage medium having stored thereon instructions which, when executed by one or more processors, configure the processors to performs a method in a Hadoop™ based distributed cluster comprising a plurality of name nodes and a plurality of data nodes and having a distributed file system, the method comprising:
- on the backup name node:
- assuming an active role to become a new active name node, upon detecting that the original name node has failed;
- flagging all of the plurality of data nodes as untrusted;
- for each data node among the plurality of data nodes;
- queuing, instead of issuing, commands intended for a data node until the data node is flagged as trusted, and
- upon receiving an acknowledgement from the data node acknowledging the assumption of the active role of the backup name node, flagging the data node as trusted;
- and
- on a respective data node:
- receiving a first command with a first transaction number from a first name node;
- receiving a second command with a second transaction number from a second name node, wherein the second transaction number is greater than the first transaction number;
- and
- sending an acknowledgment of an active role to the second name node.
Dependencies are inferred from the claim text.