Managing a global namespace for a distributed filesystem

A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for managing a global namespace for a distributed filesystem, the method comprising:

• collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises:
• storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems;
• and
• caching data for the distributed filesystem in each cloud controller;
• dynamically managing a global namespace for the distributed file system across the two or more cloud controllers by assigning distinct portions of the global namespace across the two or more of the cloud controllers, wherein each cloud controller is configured to manage write accesses for its assigned portion of the global namespace, wherein a set of namespace mappings indicate the portions of the namespace that are assigned to each cloud controller, wherein the cloud controllers collectively change the set of namespace mappings dynamically overtime based on accesses to the distributed filesystem;
• and
• receiving at an initial cloud controller a request from a client system to access a target file in the distributed filesystem;
• using a namespace mapping for the global namespace to determine a preferred cloud controller for the request by:
• determining from the set of namespace mappings that a target cloud controller manages the portion of the namespace that includes the target file;
• determining the location and current load of the target cloud controller;
• determining a local cloud controller that is co-located with the client system and the initial cloud controller, operational, and sufficiently lightly loaded to serve as the preferred cloud controller;
• determining a remote cloud controller that is lightly loaded, operational, and located at a different geographic location as well as the amount or network bandwidth between the local cluster and the different geographic location;
• and
• based on the location of the target cloud controller and determined network and load characteristics for the local cloud controller, the remote cloud controller, and the target cloud controller, choosing one of the target cloud controller, local cloud controller, and remote cloud controller to serve as the preferred cloud controller;
• and
• wherein every cloud controller for the distributed filesystem can cache the target file and provide write access to clients seeking to modify the target file regardless of which cloud controller in the distributed filesystem has been assigned to manage the portion of the global namespace that includes the target file.

A cloud controller that manages a portion of a global namespace for a distributed filesystem, comprising:

• a processor;
• a storage mechanism that stores metadata for the distributed filesystem;
• and
• a storage management mechanism;
• wherein two or more cloud controllers collectively manage the data of the distributed filesystem;
• wherein a global namespace for the distributed filesystem is dynamically managed by two or more cloud controllers;
• wherein splitting the global namespace comprises assigning distinct portions of the global namespace across the two or more cloud controllers;
• wherein each cloud controller is configured to manage write accesses for its assigned portion of the global namespace;
• wherein a set of namespace mappings indicate the portions of the namespace that are assigned to each cloud controller;
• wherein the cloud controllers collectively change the set of namespace mappings dynamically over time based on accesses to the distributed filesystem;
• wherein the cloud controller is configured to receive a request from a client system to access a target file in the distributed filesystem;
• wherein the cloud controller is further configured to use a namespace mapping for the global namespace to determine a preferred cloud controller for the request by:
• determining from the set of namespace mappings that a target cloud controller manages the portion of the namespace that includes the target file;
• determining the location and current load of the target cloud controller;
• determining a local cloud controller that is co-located with the client system and the initial cloud controller, operational, and sufficiently lightly loaded to serve as the preferred cloud controller;
• determining a remote cloud controller that is lightly loaded, operational, and located at a different geographic location as well as the amount of network bandwidth between the local cluster and the different geographic location;
• and
• based on the location of the target cloud controller and determined network and load characteristics for the local cloud controller, the remote cloud controller, and the target cloud controller, choosing one of the target cloud controller, local cloud controller, and remote cloud controller to serve as the preferred cloud controller;
• and
• wherein every cloud controller for the distributed filesystem can cache the target file and provide write access to clients seeking to modify the target file regardless of which cloud controller in the distributed filesystem has been assigned to manage the portion of the global namespace that includes the target file.