# US 11,121,913 B2 · Method for finding failing components in a large distributed storage system connectivity > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,121,913 B2 - Document ID: US11121913B2 - Status: Granted - Kind: B2 - Application number: 16/419,058 - Filed: 2019-05-22 - Published: 2020-11-26 - Granted: 2021-09-14 - Earliest claimed: 2019-05-22 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Elliot Karl Kolodner; Anna Levin - Web page: https://patentgrep.com/patent/US11121913B2 ## CPC - H04L 41/0677: Localisation of faults - H04L 41/22: comprising specially adapted graphical user interfaces [GUI] - H04L 41/5009: Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF] - H04L 43/045: for graphical visualisation of monitoring data - H04L 43/0811: by checking connectivity - H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Embodiments of the present systems and methods may provide techniques for finding failing components in a distributed storage system. For example a method may comprise measuring problems and health of a plurality of physical and logical components in a distributed storage system, the plurality of physical and logical components forming nodes of the distributed storage system, and generating a graph of the nodes organized in a plurality of hierarchical levels, generating, for each node in the graph, a score summarizing the measured problems and health of the node, determining a highest score at a highest hierarchical level of the graph and determining the associated node as a failing component at a most significant level. ## Claims ### Claim 1 A method implemented in a computer comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method comprising: measuring problems and health of a plurality of physical and logical components in a distributed storage system, the plurality of physical and logical components forming nodes of the distributed storage system, and generating a graph of the nodes organized in a plurality of hierarchical levels; generating, for each node in the graph, a score summarizing the measured problems and health of the node; determining a highest score at a highest hierarchical level of the graph by: traversing the nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level, determining a node having a score that is greater than or equal to scores of any nodes in a sub-graph of the node at any lower level, and determining that the determined node is a failing component at a most significant level; and determining an associated node as a failing component at a most significant level. ### Claim 2 (depends on 1) The method of claim 1, further comprising generating a list of additional failing components, the list ordered by a highest score of each successive node at each successive hierarchical level of the graph. ### Claim 3 (depends on 1) The method of claim 1, wherein the measuring comprises measuring metrics relating to connectivity and processing performance of nodes comprising at least one server, network device, datacenter business offering, geographic location, and the distributed storage system as a whole, and connectivity and communication performance of communication links connecting the nodes. ### Claim 4 (depends on 3) The method of claim 3, wherein the generated score for each node and link is further a function of scores of nodes and links lower in the hierarchy of the graph. ### Claim 5 (depends on 4) The method of claim 4, wherein the generated score for each node and link is further determined according to: Weight level ⁢ ⁢ n = ∑ nodes ⁢ ⁢ in ⁢ ⁢ level ⁢ ⁢ n - 1 ⁢ Weight level ⁢ ⁢ n - 1 # ⁢ nodes ⁢ ⁢ in ⁢ ⁢ level ⁢ ⁢ n - 1 . ### Claim 6 (depends on 1) The method of claim 1, wherein generating a list of additional failing components comprises: removing the sub-tree of the failing component from the graph; traversing the remaining nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level; determining a node having a score that is greater than or equal to scores of any nodes in a sub-tree of the node at any lower level; determining that the determined node is a failing component at that hierarchical level of the graph; and repeating until no nodes having scores indicating problems remain in the graph. ### Claim 7 A system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform: measuring problems and health of a plurality of physical and logical components in a distributed storage system, the plurality of physical and logical components forming nodes of the distributed storage system, and generating a graph of the nodes organized in a plurality of hierarchical levels; generating, for each node in the graph, a score summarizing the measured problems and health of the node; generating, for each node in the graph, a score summarizing the measured problems and health of the node; determining a highest score at a highest hierarchical level of the graph by: traversing the nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level, determining a node having a score that is greater than or equal to scores of any nodes in a sub-graph of the node at any lower level, and determining that the determined node is a failing component at a most significant level; and determining an associated node as a failing component at a most significant level. ### Claim 8 (depends on 7) The system of claim 7, further performing generating a list of additional failing components, the list ordered by a highest score of each successive node at each successive hierarchical level of the graph. ### Claim 9 (depends on 8) The system of claim 8, wherein the generated score for each node and link is further a function of scores of nodes and links lower in the hierarchy of the graph. ### Claim 10 (depends on 7) The system of claim 7, wherein the measuring comprises measuring metrics relating to connectivity and processing performance of nodes comprising at least one server, network device, datacenter business offering, geographic location, and the distributed storage system as a whole, and connectivity and communication performance of communication links connecting the nodes. ### Claim 11 (depends on 10) The system of claim 10, wherein the generated score for each node and link is further determined according to: Weight level ⁢ ⁢ n = ∑ nodes ⁢ ⁢ in ⁢ ⁢ level ⁢ ⁢ n - 1 ⁢ Weight level ⁢ ⁢ n - 1 # ⁢ nodes ⁢ ⁢ in ⁢ ⁢ level ⁢ ⁢ n - 1 . ### Claim 12 (depends on 7) The system of claim 7, wherein generating a list of additional failing components comprises: removing the sub-tree of the failing component from the graph; traversing the remaining nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level; determining a node having a score that is greater than or equal to scores of any nodes in a sub-tree of the node at any lower level; determining that the determined node is a failing component at that hierarchical level of the graph; and repeating until no nodes having scores indicating problems remain in the graph. ### Claim 13 A computer program product comprising a non-transitory computer readable storage having program instructions embodied therewith, the program instructions executable by a computer, to cause the computer to perform a method comprising: measuring problems and health of a plurality of physical and logical components in a distributed storage system, the plurality of physical and logical components forming nodes of the distributed storage system, and generating a graph of the nodes organized in a plurality of hierarchical levels; generating, for each node in the graph, a score summarizing the measured problems and health of the node; generating, for each node in the graph, a score summarizing the measured problems and health of the node; determining a highest score at a highest hierarchical level of the graph by: traversing the nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level, determining a node having a score that is greater than or equal to scores of any nodes in a sub-graph of the node at any lower level, and determining that the determined node is a failing component at a most significant level; and determining an associated node as a failing component at a most significant level. ### Claim 14 (depends on 13) The computer program product of claim 13, further comprising generating a list of additional failing components, the list ordered by a highest score of each successive node at each successive hierarchical level of the graph. ### Claim 15 (depends on 13) The computer program product of claim 13, wherein the measuring comprises measuring metrics relating to connectivity and processing performance of nodes comprising at least one server, network device, datacenter business offering, geographic location, and the distributed storage system as a whole, and connectivity and communication performance of communication links connecting the nodes. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the generated score for each node and link is further a function of scores of nodes and links lower in the hierarchy of the graph, wherein the generated score for each node and link is further determined according to: Weight level ⁢ ⁢ n = ∑ nodes ⁢ ⁢ in ⁢ ⁢ level ⁢ ⁢ n - 1 ⁢ Weight level ⁢ ⁢ n - 1 # ⁢ nodes ⁢ ⁢ in ⁢ ⁢ level ⁢ ⁢ n - 1 . ### Claim 17 (depends on 13) The computer program product of claim 13, wherein generating a list of additional failing components comprises: removing the sub-tree of the failing component from the graph; traversing the remaining nodes of the plurality of hierarchical levels of the graph, starting at a highest hierarchical level; determining a node having a score that is greater than or equal to scores of any nodes in a sub-tree of the node at any lower level; determining that the determined node is a failing component at that hierarchical level of the graph; and repeating until no nodes having scores indicating problems remain in the graph.