Executing a cloud command for a distributed filesystem

A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for performing a distributed-filesystem-specific action, the method comprising:

• collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises:
• storing the data for the distributed filesystem in a cloud storage system, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage system;
• maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein the metadata hierarchy is stored in the local storage device, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of the files in the distributed filesystem;
• and
• collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in the cloud storage system, wherein cloud controllers cache in their local storage devices a subset of the file data from the cloud storage system that is being actively accessed by each respective cloud controller's clients, wherein new file data received by each cloud controller from its clients is written to the cloud storage system;
• presenting a distributed-filesystem-specific capability to a client system as a file in the distributed filesystem using a file abstraction;
• receiving at a cloud controller a request to perform a distributed-filesystem-specific action in response to the file access, wherein the request comprises a cloud-aware copy operation and specifies a source file and a destination file in the distributed filesystem, wherein the cloud controller is currently not caching the data blocks for the source file;
• using the metadata hierarchy on the cloud controller to create the destination file on the cloud controller, wherein the metadata for the destination file references the same data blocks that are associated with the source file in a set of cloud files that are stored in the cloud storage system, wherein the cloud controller creates the destination file without accessing any of the data blocks or cloud files that are associated with the source file, wherein the cloud controller updates the deduplication reference counts for the data blocks of the source file to account for the newly created destination file;
• and
• distributing a metadata snapshot that includes the metadata for the destination file and the updated deduplication information to the other cloud controllers that collectively manage the distributed filesystem to notify of the creation of the destination file;
• wherein no data blocks for the source file need to be transmitted from the cloud storage system or the cloud controller for the cloud-aware copy operation, thereby substantially reducing the network bandwidth and latency associated with copying large files on the cloud controller and substantially reducing the perceived command execution time for the client system.

A cloud controller that performs a distributed-filesystem-specific action, comprising:

• a processor;
• a storage mechanism that stores metadata for the distributed filesystem;
• and
• a storage management mechanism;
• wherein two or more cloud controllers collectively manage the data of the distributed filesystem, wherein collectively managing the data comprises:
• storing the data for the distributed filesystem in a cloud storage system, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage system;
• maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein the metadata hierarchy is stored in the local storage device, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of the files in the distributed filesystem;
• and
• collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in the cloud storage system, wherein cloud controllers cache in their local storage devices a subset of the file data from the cloud storage system that is being actively accessed by each respective cloud controller's clients, wherein new file data received by each cloud controller from its clients is written to the cloud storage system;
• wherein the cloud controller is configured to present a distributed-filesystem-specific capability to a client system as a file in the distributed filesystem using a file abstraction;
• wherein the cloud controller is further configured to receive a request to perform a distributed-filesystem-specific action in response to the file access, wherein the request comprises a cloud-aware copy operation and specifies a source file and a destination file in the distributed filesystem, wherein the cloud controller is currently not caching the data blocks for the source file;
• wherein the cloud controller is further configured to:
• use the metadata hierarchy on the cloud controller to create the destination file on the cloud controller, wherein the metadata for the destination file references the same data blocks that are associated with the source file in a set of cloud files that are stored in the cloud storage system, wherein the cloud controller creates the destination file without accessing any of the data blocks or cloud files that are associated with the source file, wherein the cloud controller updates the deduplication reference counts for the data blocks of the source file to account for the newly created destination file;
• and
• distribute a metadata snapshot that includes the metadata for the destination file and the updated deduplication information to the other cloud controllers that collectively manage the distributed filesystem to notify of the creation of the destination file;
• and
• wherein no data blocks for the source file need to be transmitted from the cloud storage system or the cloud controller for the cloud-aware copy operation, thereby substantially reducing the network bandwidth and latency associated with copying large files on the cloud controller and substantially reducing the perceived command execution time for the client system.