# US 6,954,877 B2 · Fault tolerance using logical checkpointing in computing systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 6,954,877 B2
- Document ID: US6954877B2
- Status: Granted
- Kind: B2
- Application number: 09/997,877
- Filed: 2001-11-29
- Published: 2004-10-07
- Granted: 2005-10-11
- Earliest claimed: 2001-11-29
- Assignees: Agami Systems, Inc.
- Inventors: William J. Earl; Jeff Tofano
- Web page: https://patentgrep.com/patent/US6954877B2

## CPC

- G06F 11/1446: Point-in-time backing up or restoration of persistent data
- G06F 3/0601: {Interfaces specially adapted for storage systems}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system and method for checkpointing a primary computer process to a backup computer process such that if there is a failure of a primary process, the backup process can takeover without interruption. In addition, upgrades to different version of software or equipment can take place without interruption. The invention provides a lightweight checkpointing method that allows checkpointing of only external requests or messages that change the state of the service instance, thereby reducing the overhead and performance penalties. In addition, the present invention checkpoints data for primary and backups that do not share resources but are logically equivalent. All communication between the primary and backup takes places using network protocols.

## Claims

### Claim 1

A service operating on a computer system to process requests from client processes, comprising:
a primary service instance operating on the computer system, including means for receiving and processing client requests, from the client processes; and
at least one backup service instance operating on the computer system, including means for receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the primary service instance includes means for determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change; and means for communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
wherein each backup service instance includes means for communicating, to the primary service instance, acknowledgement of the determined logical requests.

### Claim 2 (depends on 1)

The service of claim 1, wherein the primary service instance includes means for committing the determined logical requests in response to the acknowledgements of the determined logical requests.

### Claim 3 (depends on 2)

The service of claim 2, wherein each backup service instance includes means for committing the results of processing the determined logical requests.

### Claim 4 (depends on 3)

The service of claim 3, further comprising: order ensuring means for ensuring that, for a sequence of determined logical requests, each backup service instance commits the results of processing the determined logical requests in the same order as the primary service instance commits the results.

### Claim 5 (depends on 4)

The service of claim 4, wherein: the order ensuring means includes means for maintaining a table of determined logical requests; and each backup service instance includes means for processing the table of determined logical requests to determine an order of committing results of processing the determined logical requests.

### Claim 6 (depends on 5)

The service of claim 5, wherein: the means for maintaining a table of determined logical requests includes means for associating sequence values with the determined logical requests; and the means for processing the table includes means for processing the sequence values.

### Claim 7 (depends on 2)

The service of claim 2, wherein: a sequence of associated client requests is a transaction; each backup service instance includes means for communicating a single acknowledgement for the client requests of the transaction; and each backup service instance includes means for committing the results of processing the client requests of the transaction as a single commitment.

### Claim 8

A service operating on a computer system to process requests from client processes, comprising:
a primary service instance operating on the computer system, including means for receiving and processing client requests, from the client processes; and
at least one backup service instance operating on the computer system, including means for receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the primary service instance includes means for determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change; means for communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance;
means for receiving an indication of a path failure between the primary service instance and a client process that occurs after a result of processing a particular client request is committed and before the client process receives the result from the service; and means for receiving the particular client request again and for providing the response to the client request without again processing the client request.

### Claim 9

A service operating on a computer system to process requests from client processes, comprising:
a primary service instance operating on the computer system, including means for receiving and processing client requests, from the client processes; and
at least one backup service instance operating on the computer system, including means for receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the primary service instance includes means for determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change; means for communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
wherein each backup service instance includes means for receiving an indication of a failure of the primary service instance, and means for notifying the client process that each backup service instance has become a new primary service instance; and the new primary service instance includes means for receiving client requests and, for client requests that have already been committed by the new primary service instance when the new primary service instance was a backup service instance, providing the committed response to the client request without again processing the request.

### Claim 10 (depends on 9)

The service of claim 9, wherein: the new primary service instance includes means for receiving client requests and, for client requests that have already been received but not committed by the new primary service instance when the new primary service instance was a backup service instance, allowing the new primary service instance to continue processing the requests.

### Claim 11 (depends on 10)

The service of claim 10, wherein: a sequence of associated client requests is a transaction; each backup service instance includes means for communicating an acknowledgement for each client request of the transaction; and each backup service instance includes means for committing the results of processing the client requests of the transaction as a single commitment.

### Claim 12 (depends on 11)

The service of claim 11, wherein: the transaction committing means of each backup service instance includes means for receiving a commitment request from the primary service instance, wherein the transaction committing means commits the results of the transaction in response to the received commitment request.

### Claim 13

A service operating on a computer system to process requests from client processes, comprising:
a primary service instance operating on the computer system, including means for receiving and processing client requests, from the client processes; and
at least one backup service instance operating on the computer system, including means for receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the primary service instance includes means for determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change; means for communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance;
means for creating an additional backup service instance, including means for replicating the logical view of the primary service instance to the additional backup service instance, wherein the means for replicating the logical view of the primary service instance to the additional backup service instance provides the determined logical requests to the additional backup service instance; and
processes a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

### Claim 14 (depends on 13)

The service of claim 13, wherein: the replicating means includes means for configuring the determining means to process client requests received while the request history table is being processed.

### Claim 15 (depends on 14)

The service of claim 14, wherein the means for configuring the determining means to process client requests received while the request history table is being processed operates in response to a termination of the request history table being processed.

### Claim 16 (depends on 15)

The service of claim 15, wherein the transitioning means includes means for configuring the primary service instance to temporarily stop accepting client requests.

### Claim 17 (depends on 16)

The service of claim 16, wherein the means for configuring the primary service instance to temporarily stop accepting client requests operates in response to a comparison of the logical view of the primary service instance to the logical view of the additional backup service instance.

### Claim 18

A service operating on a computer system to process requests from client processes, comprising:
a primary service instance operating on the computer system, and being adapted to receive and process client requests from the client processes;
at least one backup service instance operating on the computer system, and being adapted to receive and process the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the primary service instance determines which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicates with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
a mechanism for creating and operating an additional backup service instance, including a replicating mechanism for replicating the logical view of the primary service instance to the additional backup service instance, and
for providing the determined logical requests to the additional backup service instance.

### Claim 19 (depends on 18)

The service of claim 18, wherein: the replicating mechanism is further adapted to process a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

### Claim 20

A service operating on a computer system to process requests from client processes, comprising:
a primary service instance operating on the computer system, and being adapted to receive and process client requests from the client processes;
at least one backup service instance operating on the computer system, and being adapted to receive and process the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the primary service instance determines which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicates with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
a mechanism for creating and operating an additional backup service instance, including a replicating mechanism for replicating the logical view of the primary service instance to the additional backup service instance, and
for implementing the additional backup service instance to have a physical behavior different from the physical behavior of the primary service instance.

### Claim 21 (depends on 20)

The service of claim 20, further comprising: a disabling mechanism for disabling the at least one backup service instance; a mechanism for enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and a replicating mechanism replicating the logical view of the primary service instance to the at least one backup service instance.

### Claim 22 (depends on 21)

The service of claim 21, a disabling mechanism for disabling the primary service instance; and a mechanism for causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

### Claim 23 (depends on 22)

The service of claim 22, a mechanism for creating and operating a second additional backup service instance, including a replicating mechanism for replicating the logical view of the primary service instance to the second additional backup service instance.

### Claim 24

A service operating on a computer system to process requests from client processes, comprising:
a primary service instance operating on the computer system, and being adapted to receive and process client requests from the client processes;
at least one backup service instance operating on the computer system, and being adapted to receive and process the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the primary service instance determines which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicates with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
a mechanism for creating and operating an additional backup service instance, including a replicating mechanism for replicating the logical view of the primary service instance to the additional backup service instance, and
for implementing the additional backup service instance to have a physical behavior the same as the physical behavior of the primary service instance.

### Claim 25 (depends on 24)

The service of claim 24, further comprising: a disabling mechanism for disabling the at least one backup service instance; a mechanism for enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and a replicating mechanism for replicating the logical view of the primary service instance to the at least one backup service instance.

### Claim 26 (depends on 25)

The service of claim 25, further comprising: a disabling mechanism for disabling the primary service instance; and a mechanism for causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

### Claim 27 (depends on 26)

The service of claim 26, further comprising: a mechanism for creating and operating a second additional backup service instance, including a replicating mechanism for replicating the logical view of the primary service instance to the second additional backup service instance.

### Claim 28

A method of operating a computer system to process requests from client processes, comprising:
operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
wherein the step of operating each backup service instance includes communicating, to the primary service instance, acknowledgement of the determined logical requests.

### Claim 29

A method of operating a computer system to process requests from client processes, comprising:
operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance, and
committing the determined logical requests in response to the acknowledgements of the determined logical requests.

### Claim 30 (depends on 29)

The method of claim 29, wherein the step of operating each backup service instance includes committing the results of processing the determined logical requests.

### Claim 31 (depends on 30)

The method of claim 30, further comprising: ensuring that, for a sequence of determined logical requests, each backup service instance commits the results of processing the determined logical requests in the same order as the primary service instance commits the results.

### Claim 32 (depends on 31)

The method of claim 31, wherein: the order ensuring step includes maintaining a table of determined logical requests; and the step of operating each backup service instance includes processing the table of determined logical requests to determine an order of committing results of processing the determined logical requests.

### Claim 33 (depends on 32)

The method of claim 32, wherein: the step of maintaining a table of determined logical requests includes associating sequence values with the determined logical requests; and the step of processing the table includes processing the sequence values.

### Claim 34 (depends on 29)

The method of claim 29, wherein: a sequence of associated client requests is a transaction; the step of operating each backup service instance includes communicating a single acknowledgement for the client requests of the transaction; and committing the results of processing the client requests of the transaction as a single commitment.

### Claim 35

A method of operating a computer system to process requests from client processes, comprising:
operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance,
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
wherein: the step of operating the primary service instance includes receiving an indication of a path failure between the primary service instance and a client process that occurs after a result of processing a particular client request is committed and before the client process receives the result from the service, and receiving the particular client request again and providing the response to the client request without again processing the client request.

### Claim 36

A method of operating a computer system to process requests from client processes, comprising:
operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
wherein: operating each backup service instance includes receiving an indication of a failure of the primary service instance, and notifying the client process that each backup service instance has become a new primary service instance; and operating the new primary service instance includes receiving client requests and, for client requests that have already been committed by the new primary service instance when the new primary service instance was a backup service instance, providing the committed response to the client request without again processing the request.

### Claim 37 (depends on 36)

The method of claim 36, wherein: the step of operating the new primary service instance includes receiving client requests and, for client requests that have already been received but not committed by the new primary service instance when the new primary service instance was a backup service instance, allowing the new primary service instance to continue processing the requests.

### Claim 38 (depends on 37)

The method of claim 37, wherein a sequence of associated client requests is a transaction; the step of operating each backup service instance includes the step of operating each backup service instance includes communicating an acknowledgement for each client request of the transaction, and committing the results of processing the client requests of the transaction as a single commitment.

### Claim 39 (depends on 38)

The method of claim 38, wherein: committing the results of processing the client requests of a transaction for each backup service instance includes receiving a commitment request from the primary service instance, and committing the results of the transaction in response to the received commitment request.

### Claim 40

A method of operating a computer system to process requests from client processes, comprising:
operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes;
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change; and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
creating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance; wherein the step of replicating the logical view of the primary service instance to the additional backup service instance includes providing the determined logical requests to the additional backup service instance, and
processing a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

### Claim 41 (depends on 40)

The method of claim 40, wherein: the step of replicating the logical view of the primary service instance includes configuring the determining step to process client requests received while the request history table is being processed.

### Claim 42 (depends on 41)

The method of claim 41, wherein the step of configuring the determining step to process client requests received while the request history table is being processed is performed in response to a termination of the request history table being processed.

### Claim 43 (depends on 42)

The method of claim 42, wherein the transitioning step includes configuring the replacement primary service instance to temporarily stop accepting client requests.

### Claim 44 (depends on 43)

The method of claim 43, wherein the step of configuring the replacement primary service instance to temporarily stop accepting client requests is performed in response to a comparison of the logical view of the primary service instance to the logical view of the additional backup service instance.

### Claim 45

A method of operating a service on a computer system to process requests from client processes, comprising:
operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;
wherein: the step of replicating the logical view of the primary service instance to the additional backup service instance includes providing the determined logical requests to the additional backup service instance.

### Claim 46

A method of operating a service on a computer system to process requests from client processes, comprising:
operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;
wherein: the replicating step includes processing a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

### Claim 47

A method of operating a service on a computer system to process requests from client processes, comprising:
operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;
wherein: the step of replicating the logical view of the primary service instance to the additional backup service instance includes implementing the additional backup service instance to have a physical behavior different from the physical behavior of the primary service instance.

### Claim 48 (depends on 47)

The method of claim 47, further comprising: disabling the at least one backup service instance; enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and replicating the logical view of the primary service instance to the at least one backup service instance.

### Claim 49 (depends on 48)

The method of claim 48, comprising: disabling the primary service instance; and causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

### Claim 50 (depends on 49)

The method of claim 49, further comprising: creating and operating a second additional backup service instance, including replicating the logical view of the primary service instance to the second additional backup service instance.

### Claim 51

A method of operating a service on a computer system to process requests from client processes, comprising:
operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance,
wherein: the step of replicating the logical view of the primary service instance to the additional backup service instance includes implementing the additional backup service instance to have a physical behavior the same as the physical behavior of the primary service instance.

### Claim 52 (depends on 51)

The method of claim 51, further comprising: disabling the at least one backup service instance; enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and replicating the logical view of the primary service instance to the at least one backup service instance.

### Claim 53 (depends on 52)

The method of claim 52, further comprising: disabling the primary service instance; and causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

### Claim 54 (depends on 53)

The method of claim 53, further comprising: creating and operating a second additional backup service instance, including replicating the logical view of the primary service instance to the second additional backup service instance.

### Claim 55

A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:
operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
wherein the step of operating each backup service instance includes communicating, to the primary service instance, acknowledgement of the determined logical requests.

### Claim 56 (depends on 55)

The program storage device of claim 55, wherein the step of operating the primary service instance includes committing the determined logical requests in response to the acknowledgements of the determined logical requests.

### Claim 57 (depends on 56)

The program storage device of claim 56, wherein the step of operating each backup service instance includes committing the results of processing the determined logical requests.

### Claim 58 (depends on 57)

The program storage device of claim 57, further comprising: ensuring that, for a sequence of determined logical requests, each backup service instance commits the results of processing the determined logical requests in the same order as the primary service instance commits the results.

### Claim 59 (depends on 58)

The program storage device of claim 58, wherein: step order ensuring step includes maintaining a table of determined logical requests; and the step of operating each backup service instance includes processing the table of determined logical requests to determine an order of committing results of processing the determined logical requests.

### Claim 60 (depends on 59)

The program storage device of claim 59, wherein: the step of maintaining a table of determined logical requests includes associating sequence values with the determined logical requests; and the step of processing the table includes processing the sequence values.

### Claim 61 (depends on 56)

The program storage device of claim 56, wherein: a sequence of associated client requests is a transaction; the step of operating each backup service instance includes communicating a single acknowledgement for the client requests of the transaction; and the step of operating each backup service instance includes committing the results of processing the client requests of the transaction as a single commitment.

### Claim 62

A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:
operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance,
receiving an indication of a path failure between the primary service instance and a client process that occurs after a result of processing a particular client request is committed and before the client process receives the result from the service, and receiving the particular client request again and providing the response to the client request without again processing the client request.

### Claim 63

A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:
operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance;
wherein the step of operating each backup service instance includes receiving an indication of a failure of the primary service instance, and notifying the client process that each backup service instance has become a new primary service instance; and wherein the step of operating the new primary service instance includes receiving client requests and, for client requests that have already been committed by the new primary service instance when the new primary service instance was a backup service instance, providing the committed response to the client request without again processing the request.

### Claim 64 (depends on 63)

The program storage device of claim 63, wherein: the step of operating the new primary service instance includes receiving client requests and, for client requests that have already been received but not committed by the new primary service instance when the new primary service instance was a backup service instance, allowing the new primary service instance to continue processing the requests.

### Claim 65 (depends on 63)

The program storage device of claim 63, wherein: a sequence of associated client requests is a transaction; the step of operating each backup service instance includes communicating an acknowledgement for each client request of the transaction; and the step of operating each backup service instance includes committing the results of processing the client requests of the transaction as a single commitment.

### Claim 66 (depends on 65)

The program storage device of claim 65, wherein: the step of committing a transaction for each backup service instance includes receiving a commitment request from the primary service instance, wherein the transaction committing step includes committing the results of the transaction in response to the received commitment request.

### Claim 67

A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:
operating a primary service instance on the computer system, including receiving and processing client requests, from the client processes; and
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
creating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;
wherein the step of replicating the logical view of the primary service instance to the additional backup service instance includes providing the determined logical requests to the additional backup service instance, and
processing a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

### Claim 68 (depends on 67)

The program storage device of claim 67, wherein: the step of replicating the logical view of the primary service instance includes configuring the determining step to process client requests received while the request history table is being processed.

### Claim 69 (depends on 68)

The program storage device of claim 68, wherein the step of configuring the determining step to process client requests received while the request history table is being processed is performed in response to a termination of the request history table being processed.

### Claim 70 (depends on 69)

The program storage device of claim 69, wherein the transitioning step includes configuring the replacement primary service instance to temporarily stop accepting client requests.

### Claim 71 (depends on 70)

The program storage device of claim 70, wherein the step of configuring the replacement primary service instance to temporarily stop accepting client requests is performed in response to a comparison of the logical view of the primary service instance to the logical view of the additional backup service instance.

### Claim 72

A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:
operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;
wherein: the step of replicating the logical view of the primary service instance to the additional backup service instance includes providing the determined logical requests to the additional backup service instance.

### Claim 73 (depends on 72)

The program storage device of claim 72, wherein: the replicating step includes processing a request history table to ensure that logical view of the additional backup service instance is the same as the logical view of the primary service instance.

### Claim 74

A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:
operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;
wherein: the step of replicating the logical view of the primary service instance to the additional backup service instance includes implementing the additional backup service instance to have a physical behavior different from the physical behavior of the primary service instance.

### Claim 75 (depends on 74)

The program storage device of claim 74, further comprising: disabling the at least one backup service instance; enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and replicating the logical view of the primary service instance to the at least one backup service instance.

### Claim 76 (depends on 75)

The program storage device of claim 75, further comprising: disabling the primary service instance; and causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

### Claim 77 (depends on 76)

The program storage device of claim 76, further comprising: creating and operating a second additional backup service instance, including replicating the logical view of the primary service instance to the second additional backup service instance.

### Claim 78

A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform method steps for operating a computer system to process requests from client processes, said method steps comprising:
operating a primary service instance on the computer system, including receiving and processing client requests from the client processes;
operating at least one backup service instance on the computer system, including receiving and processing the client requests, wherein each backup service instance is logically equivalent to the primary service instance;
wherein the step of operating the primary service instance includes determining which external requests, when processed by the secondary service instance, are logical requests such that processing of the determined logical requests cause the external view of each backup service instance to change, and communicating with each backup service instance to provide an indication of the determined logical requests to each backup service instance; and
creating and operating an additional backup service instance, including replicating the logical view of the primary service instance to the additional backup service instance;
wherein: the step of replicating the logical view of the primary service instance to the additional backup service instance includes implementing the additional backup service instance to have a physical behavior the same as the physical behavior of the primary service instance.

### Claim 79 (depends on 78)

The program storage device of claim 78, further comprising: disabling the at least one backup service instance; enabling and operating the at least one backup service instance with an implementation that has the same physical behavior as the additional backup service instance; and replicating the logical view of the primary service instance to the at least one backup service instance.

### Claim 80 (depends on 79)

The program storage device of claim 79, further comprising: disabling the primary service instance; and causing at least one of the at least one backup service instance and the additional backup service instance to become a primary service instance.

### Claim 81 (depends on 80)

The program storage device of claim 80, further comprising: creating and operating a second additional backup service instance, including replicating the logical view of the primary service instance to the second additional backup service instance.