# US 2017/0286208 A1 · Methods and apparatuses for improving failure recovery in a distributed system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286208 A1
- Document ID: US20170286208A1
- Status: Published application
- Kind: A1
- Application number: 15/086,526
- Filed: 2016-03-31
- Published: 2017-10-05
- Earliest claimed: 2016-03-31
- Assignees: Change Healthcare LLC
- Applicants: Change Healthcare LLC
- Inventors: Albert Hives; Dominique Plante; Alex Koshta; Evan Schnell; Divya Kosaraju; Jordan Liu; Sherry Bath; John D. Schiffer; Anthony Costello
- Web page: https://patentgrep.com/patent/US20170286208A1

## CPC

- G06F 11/0793: {Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40)}
- G06F 11/0709: {in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems}
- G06F 11/0772: {Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments are disclosed for improving failure recovery in a distributed system during execution of a processing chain. In the context of a method, an example embodiment includes implementing, by the distributed system, a task coordination state machine corresponding to the processing chain. This example embodiment of the method further includes discovering, using the task coordination state machine, an error occurring during execution of the processing chain. Finally, the example embodiment of the method further includes reporting the error by a failure report generator service and recovering, by a failure recovery service, from the error. Corresponding apparatuses and computer program products are also provided.

## Claims

### Claim 1

A method for improving failure recovery in a distributed system during execution of a processing chain, the method comprising:
implementing, by the distributed system, a task coordination state machine corresponding to the processing chain;
discovering, using the task coordination state machine, an error occurring during execution of the processing chain;
reporting the error by a failure report generator service; and
recovering, by a failure recovery service, from the error.

### Claim 2 (depends on 1)

The method of claim 1, wherein the task coordination state machine comprises:
a set of states, wherein the set of states includes the error state and a state corresponding to each unit of work in the processing chain;
a set of state transitions, each of which illustrates a logical connections between two states in the set of states; and
a set of events defining changes in a context of the processing chain that correspond to each state transition.

### Claim 3 (depends on 1)

The method of claim 1, wherein discovering the error using the task coordination state machine includes:
in response to occurrence of the error,
transitioning the task coordination state machine from a particular working state into an error state, and
generating a notification based on the error.

### Claim 4 (depends on 3)

The method of claim 3, wherein the notification identifies a reason code, a time stamp, a type of the error, a severity of the error, and a criticality of the error.

### Claim 5 (depends on 1)

The method of claim 1, wherein reporting the error by the failure report generator service includes:
generating an incident based on a notification generated during discovery of the error, wherein the generated incident identifies an incident context and one or more reason codes describing one or more failures that caused the error;
storing the incident in an incident repository; and
notifying applications affected by the incident.

### Claim 6 (depends on 5)

The method of claim 5, wherein reporting the error by the failure report generator service further includes:
causing presentation, by a user device, of the incident, wherein the presentation includes a reason code associated with the incident, a summary of the incident, and information regarding how to recover from the incident; and
awaiting instructions, from the user device, to initiate a failure recovery service to recover from the incident.

### Claim 7 (depends on 5)

The method of claim 5, wherein the incident further identifies a correlation identifier that can be used to stitch together events in the system, and wherein the incident context identifies. at least one of a source unit of work, a target unit of work, and an initiating system or user that initiated processing that caused the error.

### Claim 8 (depends on 1)

The method of claim 1, wherein recovering, by the failure recovery service, from the error includes:
calling, by the failure recovery service, a resolution plan selector for selection of one or more resolvers to recover from the error;
retrieving, by the resolution plan selector, one or more resolvers responsive to one or more reason codes identified by an incident associated with the error;
executing, by a resolution plan executor, the one or more resolvers to rehydrate a context of a particular working state of the task coordination state machine that was in effect prior to occurrence of the error; and
transitioning, by the distributed system, the task coordination state machine to the particular working state.

### Claim 9

An apparatus for improving failure recovery in a distributed system during execution of a processing chain, the apparatus comprising at least one processor and at least one memory storing computer-executable instructions, that, when executed by the at least one processor, cause the apparatus to:
implement a task coordination state machine corresponding to the processing chain;
discovering, using the task coordination state machine, an error occurring during execution of the processing chain;
report the error; and
recover from the error.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the task coordination state machine comprises:
a set of states, wherein the set of states includes the error state and a state corresponding to each unit of work in the processing chain;
a set of state transitions, each of which illustrates a logical connections between two states in the set of states; and
a set of events defining changes in a context of the processing chain that correspond to each state transition.

### Claim 11 (depends on 9)

The apparatus of claim 9, wherein the computer-executable instructions, when executed by the at least one processor, cause the apparatus to discover the error using the task coordination state machine by causing the apparatus to:
in response to occurrence of the error,
transition the task coordination state machine from a particular working state into an error state, and
generate a notification based on the error.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the notification identifies a reason code, a time stamp, a type of the error, a severity of the error, and a criticality of the error.

### Claim 13 (depends on 9)

The apparatus of claim 9, wherein the computer-executable instructions, when executed by the at least one processor, cause the apparatus to report the error by causing the apparatus to:
generate an incident based on a notification generated during discovery of the error, wherein the generated incident identifies an incident context and one or more reason codes describing one or more failures that caused the error;
store the incident in an incident repository; and
notify applications affected by the incident.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the computer-executable instructions, when executed by the at least one processor, cause the apparatus to report the error by causing the apparatus to:
cause presentation, by a user device, of the incident, wherein the presentation includes a reason code associated with the incident, a summary of the incident, and information regarding how to recover from the incident; and
await instructions, from the user device, to initiate a failure recovery service to recover from the incident.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein the incident further identifies a correlation identifier that can be used to stitch together events in the system, and wherein the incident context identifies. at least one of a source unit of work, a target unit of work, and an initiating system or user that initiated processing that caused the error.

### Claim 16 (depends on 9)

The apparatus of claim 9, wherein the computer-executable instructions, when executed by the at least one processor, cause the apparatus to recover from the error by causing the apparatus to:
retrieve one or more resolvers responsive to one or more reason codes identified by an incident associated with the error;
execute the one or more resolvers to rehydrate a context of a particular working state of the task coordination state machine that was in effect prior to occurrence of the error; and
transition the task coordination state machine to the particular working state.

### Claim 17

A computer program product comprising at least one non-transitory computer-readable storage medium for improving failure recovery in a distributed system during execution of a processing chain, the at least one non-transitory computer-readable storage medium storing computer-executable instructions that, when executed, cause an apparatus to:
implement a task coordination state machine corresponding to the processing chain;
discovering, using the task coordination state machine, an error occurring during execution of the processing chain;
report the error; and
recover from the error.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein the computer-executable instructions, when executed, cause the apparatus to report the error by causing the apparatus to:
generate an incident based on a notification generated during discovery of the error, wherein the generated incident identifies an incident context and one or more reason codes describing one or more failures that caused the error;
store the incident in an incident repository; and
notify applications affected by the incident.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein the computer-executable instructions, when executed, cause the apparatus to report the error by causing the apparatus to:
cause presentation, by a user device, of the incident, wherein the presentation includes a reason code associated with the incident, a summary of the incident, and information regarding how to recover from the incident; and
await instructions, from the user device, to initiate a failure recovery service to recover from the incident.

### Claim 20 (depends on 17)

The computer program product of claim 17, wherein the computer-executable instructions, when executed, cause the apparatus to recover from the error by causing the apparatus to:
retrieve one or more resolvers responsive to one or more reason codes identified by an incident associated with the error;
execute the one or more resolvers to rehydrate a context of a particular working state of the task coordination state machine that was in effect prior to occurrence of the error; and
transition the task coordination state machine to the particular working state.