# US 2018/0321860 A1 · Data storage system tiering accounting for limited write endurance > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0321860 A1 - Document ID: US20180321860A1 - Status: Published application - Kind: A1 - Application number: 15/740,697 - Filed: 2016-12-28 - Published: 2018-11-08 - Earliest claimed: 2016-12-28 - Assignees: EMC IP Holding Company LLC - Applicants: EMC IP Holding Company LLC - Inventors: Roman Vladimirovich Marchenko; Nickolay Alexandrovich Dalmatov; Alexander Valeryevich Romanyukov - Web page: https://patentgrep.com/patent/US20180321860A1 ## CPC - G06F 3/0616: {in relation to life time, e.g. increasing Mean Time Between Failures [MTBF]} - G06F 3/0649: {Lifecycle management} - G06F 3/0685: {Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays} - G06F 11/3058: {Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations (thermal management in cooling arrangements of a computing system G06F1/206)} - G06F 11/3433: {for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029)} - G06F 12/0223: {User address space allocation, e.g. contiguous or non contiguous base addressing} - G06F 12/08: in hierarchically structured memory systems, e.g. virtual memory systems - G11C 7/04: with means for avoiding disturbances due to temperature effects - G11C 16/10: Programming or data input circuits - G11C 16/3495: {Circuits or methods to detect or delay wearout of nonvolatile EPROM or EEPROM memory devices, e.g. by counting numbers of erase or reprogram cycles, by using multiple memory areas serially or cyclically} - G06F 11/3034: {where the computing system component is a storage system, e.g. DASD based or network based (digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)} - G06F 12/0868: Data transfer between cache memory and other subsystems, e.g. storage devices or host systems - G06F 2212/1036: Life time enhancement - G06F 2212/205: Hybrid memory, e.g. using both volatile and non-volatile memory - G06F 2212/263: Network storage, e.g. SAN or NAS - G06F 2212/601: Reconfiguration of cache memory - G06F 2212/7211: Wear leveling - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G11C: STATIC STORES (semiconductor memory devices H10B) ## Abstract Techniques for performing storage tiering in a data storage system taking into account the write endurance of flash drives and the frequencies with which data are written to storage extents in a data storage system are disclosed. Such storage tiering tends to maximize data temperature of a flash tier by selecting hot extents for placement thereon, but subject to a constraint that doing so does not cause flash drives in the flash tier to operate beyond their endurance levels. ## Claims ### Claim 1 A method of storing data in a data storage system having multiple storage tiers, each storage tier composed of multiple storage drives of like performance, the method comprising: rendering storage space in each of the storage tiers as multiple slices, each slice providing an extent of contiguously addressable storage derived from a single storage tier; receiving write-endurance levels from a set of storage drives in a first storage tier, each write-endurance level from a storage drive indicating a number of writes that can be performed within a service life of the respective storage drive; calculating a maximum write rate of the first storage tier based on the write-endurance levels received from the set of storage drives in the first storage tier; measuring, for each slice in the first storage tier, (i), a slice temperature that indicates a rate of IO (Input/Output) access to data on that slice and (ii) a slice write frequency that indicates a rate at which data is written to that slice; and relocating slice data between storage tiers based on (i) the slice temperatures, (ii) the slice write frequencies, and (iii) the maximum write rate of the first storage tier. ### Claim 2 (depends on 1) The method of claim 1, wherein relocating slice data includes moving data from selected slices of the first storage tier to target slices that are not on the first storage tier. ### Claim 3 (depends on 2) The method of claim 2, further comprising: calculating a tier temperature of the first storage tier based on a sum of all slice temperatures for all slices in the first storage tier; and calculating a tier write rate based on an average of all slice write frequencies for all slices in the first storage tier, wherein relocating slice data between storage tiers increases the tier temperature of the first storage tier subject to (i) a first constraint that the tier write rate for the first storage tier does not exceed the maximum tier write rate of the first storage tier and (ii) a second constraint based on available computational resources. ### Claim 4 (depends on 3) The method of claim 3, wherein the method further comprises performing a precise tiering calculation that includes: computing a predicted tier temperature of the first storage tier for all possible configurations of slice contents located in the first storage tier; and selecting, from among all the possible configurations, a configuration that maximizes the tier temperature of the first storage tier; and wherein relocating slice data between storage tiers includes performing a tiering operation that implements the selected configuration. ### Claim 5 (depends on 4) The method of claim 4, further comprising: calculating the available computational resources in the data storage system; counting a total number of slices in the first storage tier; counting a total number of active slices across all storage tiers in the data storage system that store host-accessible data; calculating, based on the total number of slices in the first storage tier and the total number of active slices across all slices, required computational resources required to perform the precise tiering calculation within a predetermined time limit; and in response to the required computational resources being greater than the available computational resources, (i) performing an approximate tiering calculation that does not compute a tier temperature of the first storage tier for all possible configurations of slice contents and (ii) relocating slice data between storage tiers to increase the tier temperature of the first storage tier subject to a constraint that the tier write rate for the first storage tier does not exceed the maximum tier write rate of the first storage tier. ### Claim 6 (depends on 4) The method of claim 4, further comprising: counting a total number of slices in the first storage tier; counting a total number of active slices across all storage tiers in the data storage system that store host-accessible data; calculating a ratio of (i) the total number of slices in the first storage tier to (ii) the total number of active slices; and in response to the ratio exceeding a threshold, (i) performing an approximate tiering calculation that does not compute a tier temperature of the first storage tier for all possible configurations of slice contents and (ii) relocating slice data between storage tiers to increase the tier temperature of the first storage tier subject to a constraint that the tier write rate for the first storage tier does not exceed the maximum tier write rate of the first storage tier. ### Claim 7 (depends on 6) The method of claim 6, wherein the threshold is 0.5. ### Claim 8 (depends on 6) The method of claim 6, wherein the first storage tier includes multiple sub-tiers arranged by write endurance such that each sub-tier includes storage drives of like write endurance and storage drives in different sub-tiers have different write endurance, wherein each sub-tier has a respective maximum write rate based on write-endurance levels received from storage drives in that sub-tier, and wherein, when relocating slice data between storage tiers, the method further comprises moving slice data between sub-tiers based on (i) the slice write frequencies in the first storage tier, and (ii) the maximum write rates of the sub-tiers. ### Claim 9 A data storage system, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to perform a method of storing data in a data storage system having multiple storage tiers, each storage tier composed of multiple storage drives of like performance, the method comprising: rendering storage space in each of the storage tiers as multiple slices, each slice providing an extent of contiguously addressable storage derived from a single storage tier; receiving write-endurance levels from a set of storage drives in a first storage tier, each write-endurance level from a storage drive indicating a number of writes that can be performed within a service life of the respective storage drive; calculating a maximum write rate of the first storage tier based on the write-endurance levels received from the set of storage drives in the first storage tier; measuring, for each slice in the first storage tier, (i), a slice temperature that indicates a rate of IO (Input/Output) access to data on that slice and (ii) a slice write frequency that indicates a rate at which data is written to that slice; and relocating slice data between storage tiers based on (i) the slice temperatures, (ii) the slice write frequencies, and (iii) the maximum write rate of the first storage tier. ### Claim 10 (depends on 9) The data storage system of claim 9, wherein the method further comprises: calculating a tier temperature of the first storage tier based on a sum of all slice temperatures for all slices in the first storage tier; and calculating a tier write rate based on a sum of all slice write frequencies for all slices in the first storage tier, wherein relocating slice data between storage tiers increases the tier temperature of the first storage tier subject to (i) a first constraint that the tier write rate for the first storage tier does not exceed the maximum tier write rate of the first storage tier and (ii) a second constraint based on available computational resources. ### Claim 11 (depends on 10) The data storage system of claim 10, wherein the method further comprises performing a precise tiering calculation that includes: computing a predicted tier temperature of the first storage tier for all possible configurations of slice contents located in the first storage tier; and selecting, from among all the possible configurations, a configuration that maximizes the tier temperature of the first storage tier; and wherein relocating slice data between storage tiers includes performing a tiering operation that implements the selected configuration. ### Claim 12 (depends on 11) The data storage system of claim 11, wherein the method further comprises: calculating the available computational resources in the data storage system; counting a total number of slices in the first storage tier; counting a total number of active slices across all storage tiers in the data storage system that store host-accessible data; calculating, based on the total number of slices in the first storage tier and the total number of active slices across all slices, required computational resources required to perform the precise tiering calculation within a predetermined time limit; and in response to the required computational resources being greater than the available computational resources, (i) performing an approximate tiering calculation that does not compute a tier temperature of the first storage tier for all possible configurations of slice contents and (ii) relocating slice data between storage tiers to increase the tier temperature of the first storage tier subject to a constraint that the tier write rate for the first storage tier does not exceed the maximum tier write rate of the first storage tier. ### Claim 13 (depends on 11) The data storage system method of claim 11, wherein the method further comprises: counting a total number of slices in the first storage tier; counting a total number of active slices across all storage tiers in the data storage system that store host-accessible data; calculating a ratio of (i) the total number of slices in the first storage tier to (ii) the total number of active slices; and in response to the ratio exceeding a threshold, (i) performing an approximate tiering calculation that does not compute a tier temperature of the first storage tier for all possible configurations of slice contents and (ii) relocating slice data between storage tiers to increase the tier temperature of the first storage tier subject to a constraint that the tier write rate for the first storage tier does not exceed the maximum tier write rate of the first storage tier. ### Claim 14 A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a data storage system, cause the control circuitry to perform a method for storing host data, the method comprising: rendering storage space in each of the storage tiers as multiple slices, each slice providing an extent of contiguously addressable storage derived from a single storage tier; receiving write-endurance levels from a set of storage drives in a first storage tier, each write-endurance level from a storage drive indicating a number of writes that can be performed within a service life of the respective storage drive; calculating a maximum write rate of the first storage tier based on the write-endurance levels received from the set of storage drives in the first storage tier; measuring, for each slice in the first storage tier, (i), a slice temperature that indicates a rate of TO (Input/Output) access to data on that slice and (ii) a slice write frequency that indicates a rate at which data is written to that slice; and relocating slice data between storage tiers based on (i) the slice temperatures, (ii) the slice write frequencies, and (iii) the maximum write rate of the first storage tier. ### Claim 15 (depends on 14) The computer program product of claim 14, wherein the method further comprises: calculating a tier temperature of the first storage tier based on a sum of all slice temperatures for all slices in the first storage tier; and calculating a tier write rate based on an average of all slice write frequencies for all slices in the first storage tier, wherein relocating slice data between storage tiers increases the tier temperature of the first storage tier subject to (i) a first constraint that the tier write rate for the first storage tier does not exceed the maximum tier write rate of the first storage tier and (ii) a second constraint based on available computational resources. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the method further comprises performing a precise tiering calculation that includes: computing a predicted tier temperature of the first storage tier for all possible configurations of slice contents located in the first storage tier; and selecting, from among all the possible configurations, a configuration that maximizes the tier temperature of the first storage tier; and wherein relocating slice data between storage tiers includes performing a tiering operation that implements the selected configuration. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein the method further comprises: calculating the available computational resources in the data storage system; counting a total number of slices in the first storage tier; counting a total number of active slices across all storage tiers in the data storage system that store host-accessible data; calculating, based on the total number of slices in the first storage tier and the total number of active slices across all slices, required computational resources required to perform the precise tiering calculation within a predetermined time limit; and in response to the required computational resources being greater than the available computational resources, (i) performing an approximate tiering calculation that does not compute a tier temperature of the first storage tier for all possible configurations of slice contents and (ii) relocating slice data between storage tiers to increase the tier temperature of the first storage tier subject to a constraint that the tier write rate for the first storage tier does not exceed the maximum tier write rate of the first storage tier. ### Claim 18 (depends on 16) The computer program product of claim 16, wherein the method further comprises: counting a total number of slices in the first storage tier; counting a total number of active slices across all storage tiers in the data storage system that store host-accessible data; calculating a ratio of (i) the total number of slices in the first storage tier to (ii) the total number of active slices; and in response to the ratio exceeding a threshold, (i) performing an approximate tiering calculation that does not compute a tier temperature of the first storage tier for all possible configurations of slice contents and (ii) relocating slice data between storage tiers to increase the tier temperature of the first storage tier subject to a constraint that the tier write rate for the first storage tier does not exceed the maximum tier write rate of the first storage tier. ### Claim 19 (depends on 18) The computer program product of claim 18, wherein the threshold is 0.5. ### Claim 20 (depends on 18) The computer program product of claim 18, wherein the first storage tier includes multiple sub-tiers arranged by write endurance such that each sub-tier includes storage drives of like write endurance and storage drives in different sub-tiers have different write endurance, wherein each sub-tier has a respective maximum write rate based on write-endurance levels received from storage drives in that sub-tier, and wherein, when relocating slice data between storage tiers, the method further comprises moving slice data between sub-tiers based on (i) the slice write frequencies in the first storage tier, and (ii) the maximum write rates of the sub-tiers.