# US 2004/0080558 A1 · Method and apparatus for monitoring the storage of data in a computer system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2004/0080558 A1
- Document ID: US20040080558A1
- Status: Published application
- Kind: A1
- Application number: 10/281,524
- Filed: 2002-10-28
- Published: 2004-04-29
- Earliest claimed: 2002-10-28
- Assignees: EMC Corporation
- Inventors: Steven M. Blumenau; Michel Fisher
- Web page: https://patentgrep.com/patent/US20040080558A1

## CPC

- G06F 11/1435: {using file system or storage system metadata}
- G06F 11/1451: {by selection of backup contents}
- G06F 11/2064: {while ensuring consistency}
- G06F 11/2082: {Data synchronisation}
- G06F 11/3485: {for I/O devices}
- G06F 2201/88: Monitoring involving counting
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

One aspect is directed to a method for storing, in a persistent manner, a state of the storage process such that the state of the storage operation may be recovered if the storage operation is interrupted. Another aspect is directed to a method and apparatus that stores the state of a storage operation in a non-volatile memory of a computer that stores data on a storage system. Another aspect is directed to a method for recovering the state of the storage process without having to re-perform storage operations. Another aspect is directed to maintaining state information for other purposes such as monitoring access to data storage locations.

## Claims

### Claim 1

A method for use in a computer system including at least one computer and at least one storage system that is coupled to the at least one computer to store data written thereby, the method comprising acts of:
performing, by the at least one computer, at least one storage operation on a first storage location in the at least one storage system; and
storing, in a non-volatile memory of the at least one computer, information identifying a state of the at least one storage operation performed by the at least one computer.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the non-volatile memory includes a non-volatile random access memory (NVRAM) device of the at least one computer.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the act of storing further comprises an act of allocating a portion of the non-volatile memory to store the information identifying the state of the at least one storage operation before the at least one storage operation is begun, and maintaining the allocation of the portion of the non-volatile memory to store the information identifying the state of the at least one storage operation until the at least one storage operation is completed successfully.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the at least one computer is a server, and wherein the act of storing includes an act of storing the information identifying the state of the at least one storage operation in the non-volatile memory of the server.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the server is coupled to the storage system by a communication network, and wherein the act of performing at least one storage operation is performed over the communication network.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the at least one computer is a network communication system, and wherein the act of storing includes an act of storing the information identifying the state of the at least one storage operation in the non-volatile memory of the network communication system.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the act of storing further comprises an act of maintaining a count of one or more I/O operations performed on the first storage location.

### Claim 8 (depends on 7)

The method according to claim 7, further comprising an act of maintaining a count of one or more write operations performed on the first storage location.

### Claim 9 (depends on 7)

The method according to claim 7, further comprising an act of maintaining a count of one or more read operations performed on the first storage location.

### Claim 10 (depends on 8)

The method according to claim 8, further comprising an act of maintaining, separate from the count of one or more write operations, a count of one or more read operations performed on the first storage location.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the act of storing further comprises an act of maintaining a log of I/O operations performed on the first storage location.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the storage operation performed on the first location is a first write operation performed on the first storage location, and wherein the method further comprises an act of storing a status of a second write operation performed on a second storage location, wherein the second write operation performed on the second storage location corresponds to the first write operation performed on the first storage location, and wherein the method further comprises an act of determining whether:
only one of the first and second write operations completed;
both of the first and second write operations completed; and
neither of the first and second write operations completed.

### Claim 13 (depends on 12)

The method according to claim 12, further comprising acts of maintaining a first count of storage operations performed on the first storage location and maintaining a second count of storage operations performed on the second storage location, and wherein the act of determining includes comparing the first count to the second count.

### Claim 14 (depends on 12)

The method according to claim 12, wherein the act of storing further comprises an act of maintaining a log of I/O operations performed on the first storage location.

### Claim 15 (depends on 11)

The method according to claim 11, wherein the log of I/O operations comprises one or more entries each having at least one of:
a time at which data stored in the first storage location was written; and
a time at which data stored in the first storage location was read.

### Claim 16 (depends on 3)

The method according to claim 3, wherein the act of storing further comprises an act of reallocating, after the at least one storage operation is completed successfully, the portion of the non-volatile memory in which the information identifying the state of the operation was stored to store state information for at least one other storage operation on a different storage location.

### Claim 17 (depends on 1)

The method according to claim 1, wherein the at least one storage operation is part of a mirror operation performed to at least two volumes of the at least one storage system, and wherein the first storage location is located in one of the at least two volumes.

### Claim 18 (depends on 1)

The method according to claim 1, wherein the at least one storage operation is part of a migration process that migrates data between at least two volumes of the at least one storage system, and wherein the first storage location is located in one of the at least two volumes.

### Claim 19 (depends on 1)

The method according to claim 1, further comprising an act of determining, when the act of performing the at least one storage operation on the first storage location is completed, another storage location on which at least one other storage operation is to be performed, and storing a state of the at least one other storage operation in the non-volatile memory of the at least one computer.

### Claim 20

A method for recovering a state of at least one storage process in a computer system including at least one storage system that stores data written by at least one computer in the computer system, the storage process including a write operation that writes the same data to first and second data storage locations in the at least one storage system, the storage process being interrupted during the write operation, the method comprising acts of:
determining whether data stored in the first data storage location and the second data storage location is consistent; and
resuming the at least one storage process in response to the act of determining.

### Claim 21 (depends on 20)

The method according to claim 20, wherein the computer system maintains a first count of one or more storage operations performed on the first data storage location and maintains a second count of one or more storage operations performed on the second data storage location, and wherein the act of determining comprises an act of comparing the first count and the second count to determine whether the data is consistent.

### Claim 22 (depends on 20)

The method according to claim 20, wherein the computer system maintains first time stamp information identifying when one or more storage operations were performed on the first data storage location and maintains second time stamp information identifying when one or more storage operations were performed on the second data storage location, and wherein the act of determining comprises an act of comparing the first time stamp information and the second time stamp information to determine whether the data is consistent.

### Claim 23 (depends on 20)

The method according to claim 20, further comprising an act of, when it is determined that the data stored in the first and second data storage locations are not consistent, making the data stored in the first data storage location and the second data storage location consistent.

### Claim 24 (depends on 23)

The method according to claim 23, wherein the act of making further comprises acts of determining which one of the first and second data storage locations stores more recent data, and copying the more recent data to the other data storage location.

### Claim 25 (depends on 23)

The method according to claim 23, wherein the act of making further comprises an act of invalidating the data stored in the first and second data storage locations.

### Claim 26 (depends on 20)

The method according to claim 20, wherein the at least one storage process is a migration process that migrates data from the first data storage location to the second data storage location, and the act of resuming includes resuming the migration process.

### Claim 27 (depends on 20)

The method according to claim 20, wherein the act of determining includes an act of retrieving an indication of a state of the first and second storage locations.

### Claim 28 (depends on 27)

The method according to claim 27, wherein the act of retrieving includes an act of retrieving the indication from a non-volatile memory of the at least one host.

### Claim 29 (depends on 27)

The method according to claim 27, wherein the at least one host is a server and wherein the act of retrieving includes retrieving the indication from a non-volatile memory of the server.

### Claim 30

A computer readable medium encoded with instructions for execution on a computer system, the instructions when executed, perform a method for use in a computer system including at least one computer and at least one storage system that is coupled to the at least one computer to store data written thereby, the method comprising acts of:
performing, by the at least one computer, at least one storage operation on a first storage location in the at least one storage system; and
storing, in a non-volatile memory of the at least one computer, information identifying a state of the at least one storage operation performed by the at least one computer.

### Claim 31 (depends on 30)

The computer readable medium according to claim 30, wherein the non-volatile memory includes a non-volatile random access memory (NVRAM) memory device of the at least one computer.

### Claim 32 (depends on 30)

The computer readable medium according to claim 30, wherein the act of storing further comprises an act of allocating a portion of the non-volatile memory to store the information identifying the state of the at least one storage operation before the at least one storage operation is begun, and maintaining the allocation of the portion of the non-volatile memory to store the information identifying the state of the at least one storage operation until the at least one storage operation is completed successfully.

### Claim 33 (depends on 30)

The computer readable medium according to claim 30, wherein the at least one computer is a server, and wherein the act of storing includes an act of storing the information identifying the state of the at least one storage operation in the non-volatile memory of the server.

### Claim 34 (depends on 33)

The computer readable medium according to claim 33, wherein the server is coupled to the storage system by a communication network, and wherein the act of performing at least one storage operation is performed over the communication network.

### Claim 35 (depends on 30)

The computer readable medium according to claim 30, wherein the at least one computer is a network communication system, and wherein the act of storing includes an act of storing the information identifying the state of the at least one storage operation in the non-volatile memory of the network communication system.

### Claim 36 (depends on 30)

The computer readable medium according to claim 30, wherein the act of storing further comprises an act of maintaining a count of one or more I/O operations performed on the first storage location.

### Claim 37 (depends on 36)

The computer readable medium according to claim 36, further comprising an act of maintaining a count of one or more write operations performed on the first storage location.

### Claim 38 (depends on 36)

The computer readable medium according to claim 36, further comprising an act of maintaining a count of one or more read operations performed on the first storage location.

### Claim 39 (depends on 37)

The computer readable medium according to claim 37, further comprising an act of maintaining, separate from the count of one or more write operations, a count of one or more read operations performed on the first storage location.

### Claim 40 (depends on 30)

The computer readable medium according to claim 30, wherein the act of storing further comprises an act of maintaining a log of I/O operations performed on the first storage location.

### Claim 41 (depends on 1, 30)

The computer readable medium according to claim 30, wherein the storage operation on the first location is a first write operation performed on the first storage location, and wherein the method according to claim 1, wherein the storage operation performed on the first location is a first write operation performed on the first storage location, and wherein the method further comprises an act of storing a status of a second write operation performed on a second storage location, wherein the second write operation performed on the second storage location corresponds to the first write operation performed on the first storage location, and wherein the method further comprises an act of determining whether:
only one of the first and second write operations completed;
both of the first and second write operations completed; and
neither of the first and second write operations completed.

### Claim 42 (depends on 41)

The computer readable medium according to claim 41, further comprising acts of maintaining a first count of storage operations performed on the first storage location and maintaining a second count of storage operations performed on the second storage location, and wherein the act of determining includes comparing the first count to the second count.

### Claim 43 (depends on 41)

The computer readable medium according to claim 41, wherein the act of storing further comprises an act of maintaining a log of I/O operations performed on the first storage location.

### Claim 44 (depends on 43)

The computer readable medium according to claim 43, wherein the log of I/O operations comprises one or more entries each having at least one of:
a time at which data stored in the first storage location was written; and
a time at which data stored in the first storage location was read.

### Claim 45 (depends on 32)

The computer readable medium according to claim 32, wherein the act of storing further comprises an act of reallocating, after the at least one storage operation is completed successfully, the portion of the non-volatile memory in which the information identifying the state of the operation was stored to store state information for at least one other storage operation on a different storage location.

### Claim 46 (depends on 30)

The computer readable medium according to claim 30, wherein the at least one storage operation is part of a mirror operation performed to at least two volumes of the at least one storage system, and wherein the first storage location is located in one of the at least two volumes.

### Claim 47 (depends on 30)

The computer readable medium according to claim 30, wherein the at least one storage operation is part of a migration process that migrates data between at least two volumes of the at least one storage system, and wherein the first storage location is located in one of the at least two volumes.

### Claim 48 (depends on 30)

The computer readable medium according to claim 30, further comprising an act of determining, when the act of performing the at least one storage operation on the first storage location is completed, another storage location on which at least one other storage operation is to be performed, and storing a state of the at least one other storage operation in the non-volatile memory of the at least one computer.

### Claim 49

A computer readable medium encoded with instructions for execution on a computer system, the instructions, when executed, perform a method for recovering a state of at least one storage process in a computer system including at least one storage system that stores data written by at least one computer in the computer system, the storage process including a write operation that writes the same data to first and second data storage locations in the at least one storage system, the storage process being interrupted during the write operation, the method comprising acts of:
determining whether data stored in the first data storage location and the second data storage location is consistent; and
resuming the at least one storage process in response to the act of determining.

### Claim 50 (depends on 49)

The computer readable medium according to claim 49, wherein the computer system maintains a first count of one or more storage operations performed on the first data storage location and maintains a second count of one or more storage operations performed on the second data storage location, and wherein the act of determining comprises an act of comparing the first count and the second count to determine whether the data is consistent.

### Claim 51 (depends on 50)

The computer readable medium according to claim 50, wherein the computer system maintains first time stamp information identifying when one or more storage operations were performed on the first data storage location and maintains second time stamp information identifying when one or more storage operations were performed on the second data storage location, and wherein the act of determining comprises an act of comparing the first time stamp information and the second time stamp information to determine whether the data is consistent.

### Claim 52 (depends on 51)

The computer readable medium according to claim 51, further comprising an act of, when if it is determined that the data stored in the first and second data storage locations are not consistent, making the data stored in the first data storage location and the second data storage location consistent.

### Claim 53 (depends on 52)

The computer readable medium according to claim 52, wherein the act of making further comprises acts of determining which of the first and second data storage locations stores more recent data, and copying the more recent data to the other data storage location.

### Claim 54 (depends on 52)

The computer readable medium according to claim 52, wherein the act of making further comprises an act of invalidating the data stored in the first and second data storage locations.

### Claim 55 (depends on 49)

The computer readable medium according to claim 49, wherein the at least one storage process is a migration process that migrates data from the first data storage location to the second data storage location, and the act of resuming includes resuming the migration process.

### Claim 56 (depends on 49)

The computer readable medium according to claim 49, wherein the act of determining includes act of retrieving an indication of a state of the first and second storage locations.

### Claim 57 (depends on 56)

The computer readable medium according to claim 56, wherein the act of retrieving includes an act of retrieving the indication from a non-volatile memory of the at least one host.

### Claim 58 (depends on 56)

The computer readable medium according to claim 56, wherein the at least one host is a server and wherein the act of retrieving includes retrieving the indication from a non-volatile memory of the server.

### Claim 59

A method for monitoring storage of data in a computer system including at least one computer and at least one storage system that stores data in response to at least one data storage operation executed by the at least one computer, the method comprising an act of:
maintaining, for at least one data storage location of the at least one storage system, a count of a number of input/output (I/O) operations performed on the at least one data storage location.

### Claim 60 (depends on 59)

The method according to claim 59, wherein the act of maintaining comprises an act of maintaining a count of a number of read operations performed on the at the least one data storage location.

### Claim 61 (depends on 59)

The method according to claim 59, wherein the act of maintaining comprises an act of maintaining a count of one or more write operations performed on the at least one data storage location.

### Claim 62 (depends on 60)

The method according to claim 60, wherein the act of maintaining comprises an act of maintaining, separate from the count of the number of read operations, a count of a number of write operations performed on the at least one data storage location.

### Claim 63 (depends on 59)

The method according to claim 59, wherein the act of maintaining comprises an act of storing the count in non-volatile storage device of the computer system.

### Claim 64 (depends on 59)

The method according to claim 59, wherein the act of maintaining comprises an act of storing the count in the at least one computer that performs the I/O operations.

### Claim 65 (depends on 59)

The method according to claim 59, wherein the act of maintaining comprises an act of storing the count in the storage system.

### Claim 66

A method for monitoring storage of data in a computer system including at least one computer and at least one storage system that stores data in response to at least one data storage operation executed by the at least one computer, the method comprising an act of:
maintaining, for at least one data storage location of the at least one storage system, at least one timestamp that identifies a time at which at least one data storage operation was performed on the at least one data storage location.

### Claim 67 (depends on 66)

The method according to claim 66, wherein the act of maintaining further comprises an act of updating the at least one timestamp when an operation subsequent to the at least one data storage operation is performed on the data storage location.

### Claim 68 (depends on 66)

The method according to claim 66, wherein the act of maintaining comprises maintaining at least one timestamp that identifies a time that a read operation was performed on the at least one data storage location.

### Claim 69 (depends on 66)

The method according to claim 66, wherein the act of maintaining comprises maintaining at least one timestamp that identifies a time that a write operation was performed on the at least one data storage location.

### Claim 70 (depends on 69)

The method according to claim 69, wherein the act of maintaining comprises an act of maintaining separate timestamps identifying times at which a read operation and a write operation were performed on the at least one data storage location.

### Claim 71 (depends on 66)

The method according to claim 66, wherein the act of maintaining comprises an act of storing the at least one timestamp in non-volatile storage device of the computer system.

### Claim 72 (depends on 66)

The method according to claim 66, wherein the act of maintaining comprises an act of storing the at least one timestamp in the at least one computer that performs the I/O operations.

### Claim 73 (depends on 66)

The method according to claim 66, wherein the act of maintaining comprises an act of storing the at least one timestamp in the storage system.

### Claim 74

A method for monitoring storage of data in a computer system including at least one computer and at least one storage system that stores data in response to at least one data storage operation executed by the at least one computer, the method comprising acts of:
creating a first set of state information that identifies one or more accesses to a storage location of the storage system;
storing the first set of state information in a non-volatile memory location;
and, upon a failure of the computer system, recovering a state of an application based on the first set of state information.

### Claim 75 (depends on 74)

The method according to claim 74, further comprising an act of deleting the first set of state information upon a completion of the at least one data storage operation.

### Claim 76 (depends on 74)

The method according to claim 74, wherein the data storage operation is a mirroring operation.

### Claim 77 (depends on 74)

The method according to claim 74, wherein the data storage operation is a migration operation.

### Claim 78 (depends on 74)

The method according to claim 74, wherein the first set of state information is created in response to a write operation being performed on the data storage location.

### Claim 79

A computer readable medium encoded with instructions for execution in a computer system, the instructions when executed, perform a method for monitoring storage of data in a computer system including at least one computer and at least one storage system that stores data in response to at least one data storage operation executed by the at least one computer, the method comprising acts of:
maintaining, for at least one data storage location of the at least one storage system, a count of a number of input/output (I/O) operations performed on the at least one data storage location.

### Claim 80 (depends on 79)

The computer readable medium according to claim 79, wherein the act of maintaining comprises an act of maintaining a count of a number of read operations performed on the at least one data storage location.

### Claim 81 (depends on 79)

The computer readable medium according to claim 79, wherein the act of maintaining comprises an act of maintaining a count of one or more write operations performed on the at least one data storage location.

### Claim 82 (depends on 80)

The computer readable medium according to claim 80, wherein the act of maintaining comprises an act of maintaining, separate from the count of the number of read operations, a count of a number of write operations performed on the at least one data storage location.

### Claim 83 (depends on 81)

The computer readable medium according to claim 81, wherein the act of maintaining comprises an act of storing the count in non-volatile storage device of the computer system.

### Claim 84 (depends on 79)

The computer readable medium according to claim 79, wherein the act of maintaining comprises an act of storing the count in the at least one computer that performs the I/O operations.

### Claim 85 (depends on 79)

The computer readable medium according to claim 79, wherein the act of maintaining comprises an act of storing the count in the storage system.

### Claim 86

A computer readable medium encoded with instructions for execution in a computer system, the instructions when executed, perform a method for monitoring storage of data in a computer system including at least one computer and at least one storage system that stores data in response to at least one data storage operation executed by the at least one computer, the method comprising an act of:
maintaining, for at least one data storage location of the at least one storage system, at least one timestamp that identifies a time at which at least one data storage operation was performed on the at least one data storage location.

### Claim 87 (depends on 86)

The computer readable medium according to claim 86, wherein the act of maintaining further comprises an act of updating the at least one timestamp when an operation subsequent to the at least one data storage operation is performed on the data storage location.

### Claim 88 (depends on 86)

The computer readable medium according to claim 86, wherein the act of maintaining comprises maintaining at least one timestamp that identifies a time that a read operation was performed on the at least one data storage location.

### Claim 89 (depends on 86)

The computer readable medium according to claim 86, wherein the act of maintaining comprises maintaining at least one timestamp that identifies a time that a write operation was performed on the at least one data storage location.

### Claim 90 (depends on 89)

The computer readable medium according to claim 89, wherein the act of maintaining comprises an act of maintaining separate timestamps identifying times at which a read operation and a write operation were performed on the at least one data storage location.

### Claim 91 (depends on 86)

The computer readable medium according to claim 86, wherein the act of maintaining comprises an act of storing the at least one timestamp in non-volatile storage device of the computer system.

### Claim 92 (depends on 86)

The computer readable medium according to claim 86, wherein the act of maintaining comprises an act of storing the at least one timestamp in the at least one computer that performs the I/O operations.

### Claim 93 (depends on 86)

The computer readable medium according to claim 86, wherein the act of maintaining comprises an act of storing the at least one timestamp in the storage system.

### Claim 94

A computer readable medium encoded with instructions for execution in a computer system, the instructions when executed, perform a method for monitoring storage of data in a computer system including at least one computer and at least one storage system that stores data in response to at least one data storage operation executed by the at least one computer, the method comprising acts of:
creating a first set of state information that identifies one or more accesses to a storage location of the storage system;
storing the first set of state information in a non-volatile memory location;
and, upon a failure of the computer system, recovering a state of an application based on the first set of state information.

### Claim 95 (depends on 94)

The computer readable medium according to claim 94, further comprising an act of deleting the first set of state information upon a completion of the at least one data storage operation.

### Claim 96 (depends on 94)

The computer readable medium according to claim 94, wherein the data storage operation is a mirroring operation.

### Claim 97 (depends on 94)

The computer readable medium according to claim 94, wherein the data storage operation is a migration operation.

### Claim 98 (depends on 94)

The computer readable medium according to claim 94, wherein the first set of state information is created in response to a write operation being performed on the data storage location.
