Automatic node crash detection and remediation in distributed computing systems
AssigneeInternational Business Machines Corporation
InventorsKevin Allen Hughes, Juan Jose Alvarez, Malcolm S. Allen-Ware, Charles Lefurgy, Guillermo Jesus Silva
Life of this patent
Estimating the term…
Abstract
A computer-implemented method includes: detecting, by one or more processors, an indication that suggests a node has crashed, wherein the node is included in a distributed computing environment; in response to the detecting, confirming by the one or more processors whether the node has crashed by performing a set of probes on the node; and in response to the confirming that the node has crashed, initiating by the one or more processors a remediation of the node.
Claims20
A method, comprising:
- detecting, by one or more processors, an indication that suggests a node has crashed, wherein the node is included in a distributed computing environment;
- in response to the detecting, confirming by the one or more processors whether the node has crashed by performing a set of probes on the node;
- and
- in response to the confirming that the node has crashed, initiating by the one or more processors a remediation of the node, wherein the remediation comprises:
- obtaining credentials for accessing a Baseboard Management Controller (BMC) of the node;
- and changing the credentials to a predefined default.
A computer program product comprising one or more computer readable storage media having program instructions collectively stored on the one or more computer readable storage media, the program instructions executable to:
- detect a node has an unavailable status, wherein the node is included in a cluster in a distributed computing environment;
- in response to the detecting, confirm whether the node has crashed by performing a set of probes on the node;
- and
- in response to the confirming that the node has crashed, initiate a remediation of the node,
- wherein the set of probes includes:
- a first probe;
- and a second probe that determines whether a prompt of a crash kernel is present in a console of the node.
A system for automatically detecting and remediating a node crash in a distributed computing environment, the system comprising:
- one or more processors, one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by the one or more processors to:
- detect an orchestration status of nodes in a cluster;
- responsive to detecting that one of the nodes has a status of unavailable, perform a set of probes on the one of the nodes that confirm whether the one of the nodes has experienced a kernel crash;
- and
- responsive to confirming that the one of the nodes has experienced a kernel crash, automatically removing the one of the nodes from the cluster, powering down the one of the nodes, resetting the one of the nodes to a default state, and notifying an operator that the one of the nodes experienced the kernel crash.
Dependencies are inferred from the claim text.