Scrubbing disaggregated storage

A method for a node in a collective of nodes distributed across a fabric to store data for an object manager of a host in a collective of hosts distributed across the fabric, comprising:

• mounting a nonvolatile disk storing data for hosts in the collective of hosts, the nonvolatile disk having a known format for data stored on the nonvolatile disk;
• scrubbing data on the nonvolatile disk by at least performing a direct-inspection of data on the nonvolatile disk and comparing the data to first validation meta-data associated with the data, the scrubbing tolerant of stale data;
• identifying a first object stored in the nonvolatile disk that may contain corrupted data;
• determining the object manager of the host is managing storage of the first object on the node;
• sending an identification of the first object to the object manager;
• and
• responsive to the sending, receiving a second object to update the first object;
• wherein the host is configured to lock the first object, perform a host-based validation of the first object, and if the first object is corrupted, to obtain the second object from a second host storing a valid copy of the first object and instruct a remote target storing the first object to replace the first object with the second object.

A method for a first object manager to scrub all objects on a nonvolatile disk associated with a remote target without having to read all objects on the nonvolatile disk over a network communicatively coupling the first object manager, a second object manager, and the remote target, comprising:

• receiving a message from the remote target identifying a location on the nonvolatile disk of a first object that is potentially corrupted;
• getting a copy of the first object over the network;
• invalidating the first object based at least in part on
• getting a current validation meta-data for the first object,
• computing a new validation meta-data for the first object, and
• comparing the current validation meta-data and the new validation meta-data;
• requesting, based at least in part on the invalidating, a second object being a valid copy of the corrupted object from the second object manager;
• and
• instructing the remote target to update the first object with data from the second object and to unlock the first object;
• wherein the nonvolatile disk associated with the remote target stores a number of objects for the first object manager, and wherein the first object manager performs getting the copy for a fraction of the number of objects;
• and
• the fraction is less than 10% of the number of objects.

One or more non-transitory computer-readable media having instructions to provide for a node having at least one processor to store data for an object manager of a host, and in response to execution by the node, cause the node to:

• mount a nonvolatile disk storing data for hosts in a collective of hosts, the nonvolatile disk having a known format for data stored on the nonvolatile disk;
• scrub data on the nonvolatile disk by at least performing a direct-inspection of data on the nonvolatile disk and comparing the data to first validation meta-data associated with the data, the scrubbing tolerant of stale data;
• identify a first object stored in the nonvolatile disk that may contain corrupted data;
• determine the object manager of the host is managing storage of the first object on the node;
• send an identification of the first object to the object manager;
• and
• responsive to the sending, receive a second object to update the first object;
• wherein the instructions to perform the scrub include further instructions to scrub asynchronous to processing data access requests from the host;
• and
• wherein the instructions to determine the object manager include further instructions to inspect a catalog of hosts in the collective of hosts to determine which host is responsible for the first object.

One or more non-transitory computer-readable media having instructions to provide for a first object manager having at least one processor to scrub all objects on a nonvolatile disk associated with a remote target without having to read all objects on the nonvolatile disk over a network communicatively coupling the first object manager, and a second object manager, and in response to execution by the first object manager, cause the first object manager to:

• receive a message from the remote target identifying a location on the nonvolatile disk of a first object that is potentially corrupted;
• instruct the remote target to set a lock on the first object;
• get a copy of the first object over the network;
• invalidate the first object based at least in part on
• get a current validation meta-data for the first object,
• compute a new validation meta-data for the first object, and
• compare the current validation meta-data and the new validation meta-data;
• request, based at least in part on the invalidating, a second object being a valid copy of the corrupted object from the second object manager;
• and
• instruct the remote target to update the first object with data from the second object and to unlock the first object.