System and method for transparent recovery of damaged or unavailable objects in a replicated object storage system

An apparatus for object recovery in a computer system having a plurality of nodes networked together, wherein the nodes are grouped into a plurality of systems each having multiple nodes, the apparatus comprising a processor, a memory, and a read request processing module provided for each system, the read request processing module being configured to:

• receive from a client computer a read request for an object, the object having been replicated from/to a first system in which the read request processing module resides to/from at least one second system among the plurality of systems;
• if the object of the read request is available in the first system, return the object of the read request to the client computer;
• and
• if the object of the read request is not available in the first system, perform a read from replica process, the read from replica process comprising:
• if the read request utilizes HTTP protocol, retrieving the replica of the object from the second system and using the replica of the object to return the object of the read request to the client computer, and then initiating repair from replica of the object in the first system utilizing the replica of the object, and
• if the read request utilizes a protocol different from HTTP protocol, returning a notice to the client computer in order to delay a process for the read request, retrieving the replica of the object from the second system, performing repair from replica of the object in the first system utilizing the replica of the object, and then returning the repaired object to the client computer.

A non-transitory computer-readable storage medium storing a plurality of instructions for controlling a data processor in a computer system having a plurality of nodes networked together, wherein the nodes are grouped into a plurality of systems each having multiple nodes, wherein the computer-readable storage medium is provided in each system, the plurality of instructions comprising:

• instructions that cause the data processor to receive from a client computer a read request for an object, the object having been replicated from/to a first system in which the computer-readable storage medium resides to/from at least one second system among the plurality of systems;
• instructions that cause the data processor, if the object of the read request is available in the first system, to return the object of the read request to the client computer;
• and
• instructions that cause the data processor, if the object of the read request is not available in the first system, to perform a read from replica process, the read from replica process comprising:
• if the read request utilizes HTTP protocol, retrieving the replica of the object from the second system and using the replica of the object to return the object of the read request to the client computer, and then initiating repair from the replica of the object in the first system utilizing the replica of the object, and
• if the read request utilizes a protocol different from HTTP protocol, returning a notice to the client computer in order to delay the read request, retrieving the replica of the object from the second system, performing repair from replica of the object in the first system utilizing the replica of the object, and then returning the repaired object to the client computer.