Data storage system with enforced fencing

A data storage device, comprising:

• one or more processors;
• and
• a memory storing program instructions that when executed on or across the one or more processors, cause the one or more processors to:
• perform, when acting as a primary head node for a volume partition, a flush operation that causes data stored on a first head node acting as the primary head node to be written from the first head node to particular portions of mass storage devices of a data storage system, wherein performing the flush operation comprises presenting a credential or tokens issued to the first head node for accessing the particular portions of the mass storage devices;
• receive, when acting as the primary head node for the volume partition, in response to attempting to perform an additional flush operation, an indication that the tokens or credentials issued to the first head node have been superseded by tokens or credentials issued to another head node of the data storage system;
• and
• in response to receiving the indication, assume a role of secondary head node for the volume partition, wherein data written to the volume partition is replicated to the first head node from the other head node that superseded the first head node as the primary head node.

A data storage device, comprising:

• a data storage sled;
• a controller mounted in the data storage sled;
• and
• a plurality of mass storage devices mounted in the data storage sled,
• wherein the controller is configured to:
• issue a first token, in response to a head node presenting a first credential to the controller for a volume partition, wherein the first credential has not been superseded by any other credential for the volume partition presented to the controller;
• permit the head node to write data to one or more of the mass storage devices of the data storage device when presenting the issued first token with a write request;
• issue a second token that supersedes the first token, in response to another head node presenting a second credential to the controller for the volume partition that has not been superseded by any other credential for the volume partition previously presented to the controller, wherein the second credential supersedes the first credential;
• fence off access to the one or more mass storage devices, in response to the head node presenting the first token or the first credential to the controller for the volume partition that has been superseded by the second credential presented to the controller;
• and
• permit the other head node to write data to one or more of the mass storage devices of the data storage device when presenting the issued second token with a write request.