# US 2016/0253246 A1 · Auto-didacted hierarchical failure recovery for remote access controllers

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0253246 A1
- Document ID: US20160253246A1
- Status: Published application
- Kind: A1
- Application number: 14/632,685
- Filed: 2015-02-26
- Published: 2016-09-01
- Earliest claimed: 2015-02-26
- Assignees: Dell Products L.P.
- Applicants: Dell Products, L.P.
- Inventors: Alex Chunghen CHOW; Pushkala IYER; Yogesh PRABHAKAR Kulkarni
- Web page: https://patentgrep.com/patent/US20160253246A1

## CPC

- G06F 11/1469: {Backup restoration techniques}
- G06F 11/0706: {the processing taking place on a specific hardware platform or in a specific software environment}
- G06F 11/0793: {Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40)}
- G06F 11/1464: {for networked environments}
- G06F 11/1471: involving logging of persistent data for recovery
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method and an information handling system (IHS) provide hierarchical failure recovery for firmware components of the information handling system. According to one aspect, a hierarchical recovery and learning module (HRLM) detects firmware component failure. The HRLM initiates a hierarchical failure recovery by executing recovery sequences from a set of hierarchically ordered recovery sequences. The HRLM determines whether the detected failure was corrected by executing the recovery sequence. If the HRLM further determines that the failure is not corrected by executing the set of hierarchically ordered recovery sequences, the HRLM dynamically generates a new recovery sequence(s) using commands selected from at least one of the previously executed recovery sequences and executes the new recovery sequence(s). If the failure is corrected by a new recovery sequence, the HRLM forwards the particular recovery sequence to a customer support site for use by other systems in addressing similar or identical failures.

## Claims

### Claim 1

A method of hierarchical self-healing and auto-didactic recovery for firmware components of an information handling system, the method comprising:
detecting a failure of a firmware component in the information handling system;
initiating a hierarchical failure recovery by executing at least one recovery sequence from a set of hierarchically ordered recovery sequences, corresponding to the detected failure;
following execution of a recovery sequence, performing, before executing a next recovery sequence, an evaluating to determine whether the detected failure was corrected by executing the recovery sequence;
in response to the failure not being corrected by executing the set of hierarchically ordered recovery sequences, dynamically generating at least one new recovery sequence using commands selected from at least one of the previously executed recovery sequences in order to extend the set of hierarchically ordered recovery sequences; and
executing the at least one new recovery sequence.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
updating, based on performed evaluations, one or more metrics respectively associated with a success rate of the one or more executed recovery sequences relative to each other in resolving the detected failure, wherein the one or more metrics are utilized to provide an updated set of hierarchically ordered recovery sequences appropriate for addressing a future failure.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
in response to the failure being corrected by executing the recovery sequence, dynamically initiating a learning mechanism that: records information about a respective failure and at least one corresponding recovery sequence; enables retrieval of the recorded information for report collection; and forwards the particular recovery sequence to a customer support site which subsequently provides the recovery sequence for download to other systems.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
retrieving the set of hierarchically ordered recovery sequences;
selecting a portion of commands from the at least one of the previously executed recovery sequences to create the at least one new sequence;
selectively ordering the executable commands corresponding to each of the at least one new sequence, respectively, to provide at least one corresponding series of executable commands; and
wherein each of the at least one series of executable commands differ from any other series of executable commands with respect to at least one of (a) constituent executable commands and (b) an ordering of the constituent executable commands, wherein each of the at least one series of executable commands is a corresponding new recovery sequence.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
respectively ordering selected portions of the executable commands by applying one or more metrics associated with at least one of success rates of various recovery sequences having executable commands, a type of failure of the firmware component, and a priority associated with the identified failure.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
verifying whether the detected failure has been corrected by executing a new recovery sequence; and
in response to verifying that the detected failure has been corrected:
storing information associating the specific firmware failure with the new recovery sequence that corrected the failure; and
publishing an association of the detected failure and the new recovery sequence to a support site to share via download with other information handling systems.

### Claim 7 (depends on 1)

The method of claim 1, wherein the recovery sequences are hierarchically ordered based on inter-layer system component dependencies, where various layers represent respective hierarchical levels.

### Claim 8 (depends on 3)

The method of claim 3, further comprising:
in response to establishing that the new recovery sequence, having a specific series and group of commands, corrects the failure, recording information identifying the new recovery sequence associated with the detected failure of the firmware component, and which includes a type of failure of the firmware component and a hierarchical level of the correction.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
detecting a failure of a firmware component further comprises inserting at least one software assertion in the firmware component, wherein the software assertion can determine whether the firmware component is working properly; and
once a failure is detected, the method further comprises:
identifying whether at least one hierarchical failure recovery sequence is available for that detected failure; and
generating an output indicating that the failure has been detected.

### Claim 10 (depends on 9)

The method of claim 9, wherein generating an output indicating that a failure has been detected comprises one or more of:
displaying a message on a screen of the information handling system indicating that a failure has been detected and the information handling system is entering in the self-healing mode; and
activating an LED that indicates that the information handling system is in the self-healing mode.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
in response to not being able to identify a hierarchical failure recovery sequence for the detected failure, performing one or more of:
prompting a user to manually select a hierarchical failure recovery sequence from a FFRSM database that may correct the failure;
prompting the user to select an information handling system shutdown;
displaying a message to indicate that the failure is not recoverable and a new hierarchical failure recovery sequence is required.

### Claim 12 (depends on 1)

The method of claim 1, wherein searching for the one or more recovery sequences that corresponds with the failure further comprises:
identifying the recovery sequences within local storage; and
in response to not being able to identify the recovery sequences in local storage, searching for the recovery sequences that correspond to the failure from a support site, downloading a found recovery sequence at the support site, and executing the found recovery sequence at the IHS.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
in response to a detected failure and not having access to an appropriate recovery sequence, placing the IHS into a self-healing mode;
said placing the IHS within a self-healing mode further comprises:
initiating an orderly shutdown of the information handling system; and
wherein when the information handling system cannot initiate an orderly shutdown, the information handling system is forced to shut down.

### Claim 14

An information handling system comprising:
a central unit processor (CPU);
at least one component that operates based on execution of a corresponding firmware component;
a local storage facility that stores executable firmware utilized to operate the at least one component; and
a hierarchical recovery and learning system/module comprising:
a failure detection mechanism that detects a failure of a firmware component in the information handling system; and
a hierarchical failure recovery mechanism that:
initiates a hierarchical failure recovery by executing at least one recovery sequence from a set of hierarchically ordered recovery sequences, corresponding to the detected failure;
following execution of a recovery sequence, performing, before executing a next recovery sequence, an evaluating to determine whether the detected failure was corrected by executing the recovery sequence;
in response to the failure not being corrected by executing the set of hierarchically ordered recovery sequences, dynamically generating at least one new recovery sequence using commands selected from at least one of the previously executed recovery sequences in order to extend the set of hierarchically ordered recovery sequences; and
executing the at least one new recovery sequence.

### Claim 15 (depends on 14)

The IHS of claim 14, wherein the hierarchical failure recovery mechanism:
updates, based on performed evaluations, one or more metrics respectively associated with a success rate of the one or more executed recovery sequences relative to each other in resolving the detected failure, wherein the one or more metrics are utilized to provide an updated set of hierarchically ordered recovery sequences appropriate for addressing a future failure.

### Claim 16 (depends on 14)

The IHS of claim 14, wherein the hierarchical failure recovery mechanism:
in response to the failure being corrected by executing the recovery sequence, dynamically initiates a learning mechanism that: records information about a respective failure and at least one corresponding recovery sequence; enables retrieval of the recorded information for report collection; and forwards the particular recovery sequence to a customer support site which subsequently provides the recovery sequence for download to other systems.

### Claim 17 (depends on 14)

The IHS of claim 14, wherein the hierarchical failure recovery mechanism:
retrieves the set of hierarchically ordered recovery sequences;
selects a portion of commands from the at least one of the previously executed recovery sequences to create the at least one new sequence;
selectively orders the executable commands corresponding to each of the at least one new sequence, respectively, to provide at least one corresponding series of executable commands; and
wherein each of the at least one series of executable commands differ from any other series of executable commands with respect to at least one of (a) constituent executable commands and (b) an ordering of the constituent executable commands, wherein each of the at least one series of executable commands is a corresponding new recovery sequence.

### Claim 18 (depends on 17)

The IHS of claim 17, wherein the hierarchical failure recovery mechanism:
respectively orders selected portions of the executable commands by applying one or more metrics associated with at least one of success rates of various recovery sequences having executable commands, a type of failure of the firmware component, and a priority associated with the identified failure.

### Claim 19 (depends on 14)

The IHS of claim 14, wherein the hierarchical failure recovery mechanism:
verifies whether the detected failure has been corrected by executing a new recovery sequence; and
in response to verifying that the detected failure has been corrected:
stores information associating the specific firmware failure with the new recovery sequence that corrected the failure; and
publishes an association of the detected failure and the new recovery sequence to a support site to share via download with other information handling systems.

### Claim 20 (depends on 14)

The IHS of claim 14, wherein the recovery sequences are hierarchically ordered based on inter-layer system component dependencies, where various layers represent respective hierarchical levels.

### Claim 21 (depends on 17)

The method of claim 17, wherein the hierarchical failure recovery mechanism:
in response to establishing that the new recovery sequence, having a specific series and group of commands, corrects the failure, recording information identifying the new recovery sequence associated with the detected failure of the firmware component, and which includes a type of failure of the firmware component and a hierarchical level of the correction.

### Claim 22 (depends on 14)

The IHS of claim 14, wherein:
detecting a failure of a firmware component further comprises inserting at least one software assertion in the firmware component, wherein the software assertion can determine whether the firmware component is working properly; and
once a failure is detected, the method further comprises:
identifying whether at least one hierarchical failure recovery sequence is available for that detected failure; and
generating an output indicating that the failure has been detected.

### Claim 23 (depends on 22)

The IHS of claim 22, wherein generating an output indicating that a failure has been detected comprises one or more of:
displaying a message on a screen of the information handling system indicating that a failure has been detected and the information handling system is entering in the self-healing mode; and
activating an LED that indicates that the information handling system is in the self-healing mode.

### Claim 24 (depends on 23)

The IHS of claim 23, wherein the hierarchical failure recovery mechanism:
in response to not being able to identify a hierarchical failure recovery sequence for the detected failure, performs one or more of:
prompting a user to manually select a hierarchical failure recovery sequence from a FFRSM database that may correct the failure;
prompting the user to select an information handling system shutdown; and
displaying a message to indicate that the failure is not recoverable and a new hierarchical failure recovery sequence is required.

### Claim 25 (depends on 14)

The IHS of claim 14, wherein searching for the one or more recovery sequences that corresponds with the failure further comprises:
identifying the recovery sequences within local storage; and
in response to not being able to identify the recovery sequences in local storage, searching for the recovery sequences that correspond to the failure from a support site, downloading a found recovery sequence at the support site, and executing the found recovery sequence at the IHS.

### Claim 26 (depends on 14)

The method of claim 14, wherein the hierarchical failure recovery mechanism:
in response to a detected failure and not having access to an appropriate recovery sequence, places the IHS into a self-healing mode;
said placing the IHS into the self-healing mode further comprises:
initiating an orderly shutdown of the information handling system; and
wherein when the information handling system cannot initiate an orderly shutdown, the information handling system is forced to shut down.