Programmatically choosing preferred storage parameters for files in large-scale distributed storage systems based on desired file reliability or availability
AssigneeGoogle Inc.
InventorMurray M. STOKELY
Life of this patent
Estimating the term…
Abstract
Methods to determine and automatically recommend or adjust configuration parameters for storing files in large-scale distributed storage systems are disclosed. These methods may receive file metadata and trace data that allows the system to identify file access patterns. Additionally, the methods may receive information about distributed storage systems in a datacenter. This information can be used to choose storage parameters on a per-file basis for storing files.
Claims48
A method for programmatically determining preferred storage parameters for files stored in a large-scale distributed storage system, the method comprising:
- receiving a desired file reliability or availability requirement;
- receiving system characteristics which include statistics of system outages in a datacenter;
- predicting future failures of the storage system based on system outage statistics;
- receiving file encoding or replication information for at least one file in the storage system;
- predicting file reliability and maintenance costs based on past system failures and file encoding and replication information;
- and
- determining preferred storage parameters for at least one file to minimize cost while achieving the desired file reliability or availability based on the received requirement.
A method for programmatically determining preferred storage parameters for files stored in a large-scale distributed storage system, the method comprising:
- receiving a desired file reliability or availability requirement;
- receiving system characteristics which include information regarding the storage system, off-site backups for the storage system, and file access patterns for files stored in the storage system;
- determining historical reliability of the system, reliability of off-site backups for the storage system, and reliability of files stored using each possible file encoding or replication level in the system;
- predicting the future reliability of the storage system based on the determined reliability of the storage system;
- predicting the future reliability of files stored in the storage system for each file encoding or replication level;
- determining the probability of losing both the large-scale system and the off-site storage system backups using the predicted reliability of the storage system and the off-site backups;
- and
- determining a preferred storage parameter for at least one file based on the determined probability of losing both the storage system and the off-site backups, the predicted reliability of each encoding or replication level, and the received desired file reliability or availability requirement.
A system for programmatically determining preferred storage parameters for files stored in a large-scale distributed storage system, the system comprising:
- one or more processing devices and
- one or more storage devices storing instructions that, when executed by the one or more processing devices, cause the one or more processing devices to:
- receive a desired file reliability or availability requirement;
- receive system characteristics which include statistics of system outages in a datacenter;
- predict future failures of the storage system based on system outage statistics;
- receive file encoding or replication information for at least one file in the storage system;
- predict file reliability and maintenance costs based on past system failures and file encoding and replication information;
- and
- determine preferred storage parameters for at least one file to minimize cost while achieving the desired file reliability or availability based on the received requirement.
A system for programmatically determining preferred storage parameters for files stored in a large-scale distributed storage system, the system comprising:
- one or more processing devices and
- one or more storage devices storing instructions that, when executed by the one or more processing devices, cause the one or more processing devices to:
- receive a desired file reliability or availability requirement;
- receive system characteristics which include information regarding the storage system, off-site backups for the storage system, and file access patterns for files stored in the storage system;
- determine historical reliability of the system, reliability of off-site backups for the storage system, and reliability of files stored using each possible file encoding or replication level in the system;
- predict the future reliability of the storage system based on the determined reliability of the storage system;
- predict the future reliability of files stored in the storage system for each file encoding or replication level;
- determine the probability of losing both the large-scale system and the off-site storage system backups using the predicted reliability of the storage system and the off-site backups;
- and
- determine a preferred storage parameter for at least one file based on the determined probability of losing both the storage system and the off-site backups, the predicted reliability of each encoding or replication level, and the received desired file reliability or availability requirement.
Dependencies are inferred from the claim text.