# US 2021/0240768 A1 · Reliably maintaining strict consistency in cluster wide state of opened files in a distributed file system cluster exposing a global namespace > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240768 A1 - Document ID: US20210240768A1 - Status: Published application - Kind: A1 - Application number: 16/782,426 - Filed: 2020-02-05 - Published: 2021-08-05 - Earliest claimed: 2020-02-05 - Assignees: EMC IP Holding Company LLC - Applicants: EMC IP Holding Company LLC - Inventors: Vishal Chandra Tiwary; Xiaobing Zhang; Abhishek Rajimwale - Web page: https://patentgrep.com/patent/US20210240768A1 ## CPC - G06F 16/906: Clustering; Classification - G06F 16/11: File system administration, e.g. details of archiving or snapshots (error detection or correction of the data by redundancy in operations G06F11/14) - G06F 16/182: Distributed file systems - G06F 16/9014: {hash tables} - G06F 16/907: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - H04L 67/141: Setup of application sessions (admission control or resource allocation in data switching networks H04L47/70) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Different logical partitions representing parts of a distributed file system global namespace are hosted on some cluster nodes, e.g., metadata nodes. File content and shadow logical partitions corresponding to the different logical partitions are hosted on other nodes, e.g., data nodes. Each file is associated with a metadata and data node. TCP links are established between nodes. Upon opening files, a file manager server session is generated between each pair of nodes associated with the open files to track open states and is recorded in a mapping table. The mapping table identifies each open file and associated nodes. When a metadata or data node of a particular pair of nodes associated with an open file becomes unavailable, the mapping table is consulted to identify another of the metadata or data node associated with the open file. Crash recovery protocols are performed on the other of the metadata or data node. ## Claims ### Claim 1 A method of maintaining consistency in a distributed file system of a cluster for a plurality of clients that access files via a stateful protocol comprising: hosting different logical partitions representing parts of a global namespace on some nodes of the cluster, and file content and shadow logical partitions corresponding to the different logical partitions on other nodes, the some nodes being metadata nodes and the other nodes being data nodes, each file thereby being associated with a pair of nodes comprising a metadata and data node; establishing Transmission Control Protocol (TCP) links between the metadata and data nodes; upon opening one or more files in response to file system operations requested by the clients, generating a file manager server session between each pair of nodes associated with the files to track open states of the files; recording each file manager server session in a mapping table, the mapping table thereby identifying each open file, and each particular pair of nodes associated with each open file; detecting that one of a metadata or data node of a particular pair of nodes associated with an open file has become unavailable because a TCP link between the particular pair of nodes has broken, another of the metadata or data node remaining available; consulting the mapping table to identify the other of the metadata or data node, and the open file; and performing crash recovery protocols on the other of the metadata or data node associated with the open file while not performing the crash recovery protocols for the open file on any other available node in the cluster. ### Claim 2 (depends on 1) The method of claim 1 further comprising: allowing metadata associated with the open file to become stale on other metadata nodes of the cluster that are not the metadata node of the particular pair of nodes, thereby restricting strict consistency of the open file to be between the metadata node and the data node of the particular pair of nodes. ### Claim 3 (depends on 1) The method of claim 1 further comprising: receiving a first request from a first client to open a first file associated with a first pair of nodes, the first pair of nodes comprising a first metadata node and a first data node; creating, in response to the first request, a first file manager server session and a first file handle; tagging the first file with the first file manager server session; calculating a first verifier based on the first file manager server session and a server identifier, the server identifier identifying at least one of the first metadata node or the first data node of the first pair of nodes; assigning the first verifier to the first file handle; and sending the first file handle and the first verifier to the first client, wherein the first client uses the first file handle and the first verifier in requesting subsequent file system operations on the first file. ### Claim 4 (depends on 3) The method of claim 3 further comprising: receiving from the first client, in conjunction with a subsequent file system operation requested on the first file, the first file handle and the first verifier; determining that the first verifier is old; and based on the determination that the first verifier is old, returning an error to the first client indicating that the subsequent file system operation requested on the first file cannot be performed because the subsequent file system operation has been requested on a file that is stale. ### Claim 5 (depends on 1) The method of claim 1 further comprising: receiving at a first metadata node a client request to access a file, the file being associated with a second metadata node that is paired with a second data node; and redirecting the client request to the second metadata node. ### Claim 6 (depends on 1) The method of claim 1 wherein the performing crash recovery protocols comprises: cleaning-up a file manager server session associated with the particular pair of nodes; maintaining, during the clean-up, a stale session hash; and placing the open file into the stale session hash to allow a new file manager server session to be re-established once the one of the unavailable metadata node or data node of the particular pair of nodes becomes available while the file manager server session is still in a process of being cleaned-up. ### Claim 7 (depends on 1) The method of claim 1 wherein a logical partition comprises an Mtree. ### Claim 8 A system for maintaining consistency in a distributed file system of a cluster for a plurality of clients that access files via a stateful protocol, the system comprising: a processor; and memory configured to store one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of: hosting different logical partitions representing parts of a global namespace on some nodes of the cluster, and file content and shadow logical partitions corresponding to the different logical partitions on other nodes, the some nodes being metadata nodes and the other nodes being data nodes, each file thereby being associated with a pair of nodes comprising a metadata and data node; establishing Transmission Control Protocol (TCP) links between the metadata and data nodes; upon opening one or more files in response to file system operations requested by the clients, generating a file manager server session between each pair of nodes associated with the files to track open states of the files; recording each file manager server session in a mapping table, the mapping table thereby identifying each open file, and each particular pair of nodes associated with each open file; detecting that one of a metadata or data node of a particular pair of nodes associated with an open file has become unavailable because a TCP link between the particular pair of nodes has broken, another of the metadata or data node remaining available; consulting the mapping table to identify the other of the metadata or data node, and the open file; and performing crash recovery protocols on the other of the metadata or data node associated with the open file while not performing the crash recovery protocols for the open file on any other available node in the cluster. ### Claim 9 (depends on 8) The system of claim 8 wherein the processor further carries out the steps of: allowing metadata associated with the open file to become stale on other metadata nodes of the cluster that are not the metadata node of the particular pair of nodes, thereby restricting strict consistency of the open file to be between the metadata node and the data node of the particular pair of nodes. ### Claim 10 (depends on 8) The system of claim 8 wherein the processor further carries out the steps of: receiving a first request from a first client to open a first file associated with a first pair of nodes, the first pair of nodes comprising a first metadata node and a first data node; creating, in response to the first request, a first file manager server session and a first file handle; tagging the first file with the first file manager server session; calculating a first verifier based on the first file manager server session and a server identifier, the server identifier identifying at least one of the first metadata node or the first data node of the first pair of nodes; assigning the first verifier to the first file handle; and sending the first file handle and the first verifier to the first client, wherein the first client uses the first file handle and the first verifier in requesting subsequent file system operations on the first file. ### Claim 11 (depends on 10) The system of claim 10 wherein the processor further carries out the steps of: receiving from the first client, in conjunction with a subsequent file system operation requested on the first file, the first file handle and the first verifier; determining that the first verifier is old; and based on the determination that the first verifier is old, returning an error to the first client indicating that the subsequent file system operation requested on the first file cannot be performed because the subsequent file system operation has been requested on a file that is stale. ### Claim 12 (depends on 8) The system of claim 8 wherein the processor further carries out the steps of: receiving at a first metadata node a client request to access a file, the file being associated with a second metadata node that is paired with a second data node; and redirecting the client request to the second metadata node. ### Claim 13 (depends on 8) The system of claim 8 wherein the processor further carries out the steps of: cleaning-up a file manager server session associated with the particular pair of nodes; maintaining, during the clean-up, a stale session hash; and placing the open file into the stale session hash to allow a new file manager server session to be re-established once the one of the unavailable metadata node or data node of the particular pair of nodes becomes available while the file manager server session is still in a process of being cleaned-up. ### Claim 14 (depends on 8) The system of claim 8 wherein a logical partition comprises an Mtree. ### Claim 15 A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method for maintaining consistency in a distributed file system of a cluster for a plurality of clients that access files via a stateful protocol, the method comprising: hosting different logical partitions representing parts of a global namespace on some nodes of the cluster, and file content and shadow logical partitions corresponding to the different logical partitions on other nodes, the some nodes being metadata nodes and the other nodes being data nodes, each file thereby being associated with a pair of nodes comprising a metadata and data node; establishing Transmission Control Protocol (TCP) links between the metadata and data nodes; upon opening one or more files in response to file system operations requested by the clients, generating a file manager server session between each pair of nodes associated with the files to track open states of the files; recording each file manager server session in a mapping table, the mapping table thereby identifying each open file, and each particular pair of nodes associated with each open file; detecting that one of a metadata or data node of a particular pair of nodes associated with an open file has become unavailable because a TCP link between the particular pair of nodes has broken, another of the metadata or data node remaining available; consulting the mapping table to identify the other of the metadata or data node, and the open file; and performing crash recovery protocols on the other of the metadata or data node associated with the open file while not performing the crash recovery protocols for the open file on any other available node in the cluster. ### Claim 16 (depends on 15) The computer program product of claim 15 wherein the method further comprises: allowing metadata associated with the open file to become stale on other metadata nodes of the cluster that are not the metadata node of the particular pair of nodes, thereby restricting strict consistency of the open file to be between the metadata node and the data node of the particular pair of nodes. ### Claim 17 (depends on 15) The computer program product of claim 15 wherein the method further comprises: receiving a first request from a first client to open a first file associated with a first pair of nodes, the first pair of nodes comprising a first metadata node and a first data node; creating, in response to the first request, a first file manager server session and a first file handle; tagging the first file with the first file manager server session; calculating a first verifier based on the first file manager server session and a server identifier, the server identifier identifying at least one of the first metadata node or the first data node of the first pair of nodes; assigning the first verifier to the first file handle; and sending the first file handle and the first verifier to the first client, wherein the first client uses the first file handle and the first verifier in requesting subsequent file system operations on the first file. ### Claim 18 (depends on 17) The computer program product of claim 17 wherein the method further comprises: receiving from the first client, in conjunction with a subsequent file system operation requested on the first file, the first file handle and the first verifier; determining that the first verifier is old; and based on the determination that the first verifier is old, returning an error to the first client indicating that the subsequent file system operation requested on the first file cannot be performed because the subsequent file system operation has been requested on a file that is stale. ### Claim 19 (depends on 15) The computer program product of claim 15 wherein the method further comprises: receiving at a first metadata node a client request to access a file, the file being associated with a second metadata node that is paired with a second data node; and redirecting the client request to the second metadata node. ### Claim 20 (depends on 15) The computer program product of claim 15 wherein the method further comprises: cleaning-up a file manager server session associated with the particular pair of nodes; maintaining, during the clean-up, a stale session hash; and placing the open file into the stale session hash to allow a new file manager server session to be re-established once the one of the unavailable metadata node or data node of the particular pair of nodes becomes available while the file manager server session is still in a process of being cleaned-up.