Distributing data for a distributed filesystem across multiple cloud storage systems

A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for distributing data for a distributed filesystem across multiple cloud storage systems, the method comprising:

• collectively managing the data of the distributed filesystem using two or more cloud controllers by:
• collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients can only access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in a first remote cloud storage system using cloud files, wherein each cloud controller caches a subset of the file data from the first remote cloud storage system that is being actively accessed by that cloud controller's respective clients, wherein all new file data received by each cloud controller from its clients is written to the first remote cloud storage system via the receiving cloud controller;
• maintaining at each cloud controller a copy of the complete metadata for all of the files stored in the distributed filesystem, wherein each cloud controller communicates any changes to the metadata for the distributed filesystem to the full set of cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of each file in the distributed filesystem;
• upon receiving in a cloud controller new file data from a client, storing the new file data for the distributed filesystem in the first remote cloud storage system, wherein the new file data includes two or more new files that are stored in a cloud file, wherein the cloud file is sent from the cloud controller to the first remote cloud storage system as part of an incremental data snapshot;
• and
• upon receiving confirmation that the cloud file has been successfully stored in the first remote cloud storage system, sending from the cloud controller an incremental metadata snapshot that includes new metadata for the distributed filesystem that describes the new file data and links to the cloud file, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;
• and
• distributing data stored in the distributed filesystem across the first remote cloud storage system and a second remote cloud storage system to optimize at least one of performance and cost of the distributed filesystem.

A cloud controller that distributes data for a distributed filesystem across multiple cloud storage systems, comprising:

• a processor;
• a storage mechanism that stores metadata for the distributed filesystem;
• and
• a storage management mechanism;
• wherein two or more cloud controllers collectively manage the data of the distributed filesystem, wherein collectively managing the data comprises:
• collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients can only access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in a first remote cloud storage system using cloud files, wherein each cloud controller caches a subset of the file data from the first remote cloud storage system that is being actively accessed by that cloud controller's respective clients, wherein all new file data received by each cloud controller from its clients is written to the first remote cloud storage system via the receiving cloud controller;
• maintaining at each cloud controller a copy of the complete metadata for all of the files stored in the distributed filesystem, wherein each cloud controller communicates any changes to the metadata for the distributed filesystem to the full set of cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of each file in the distributed filesystem;
• upon receiving in the cloud controller new file data from a client, storing the new file data for the distributed filesystem in the first remote cloud storage system, wherein the cloud file is sent from the cloud controller to the first remote cloud storage system as part of an incremental data snapshot;
• and
• upon receiving confirmation that the cloud file has been successfully stored in the first remote cloud storage system, sending from the cloud controller an incremental metadata snapshot that includes new metadata for the distributed filesystem that describes the new file data and links to the cloud file, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;
• and
• wherein the storage management mechanism is further configured to distribute data stored in the distributed filesystem across the first remote cloud storage system and a second remote cloud storage system to optimize at least one of performance and cost of the distributed filesystem.