# US 10,372,525 B2 · Method and device for determining causes of performance degradation for storage systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,525 B2 - Document ID: US10372525B2 - Status: Granted - Kind: B2 - Application number: 15/381,596 - Filed: 2016-12-16 - Published: 2017-06-29 - Granted: 2019-08-06 - Earliest claimed: 2015-12-23 - Assignees: EMC IP Holding Company LLC - Applicants: Emc IP Holding Company LLC - Inventors: Junping Frank Zhao; Yu Cao; Sanping Li - Web page: https://patentgrep.com/patent/US10372525B2 ## CPC - G06F 11/079: {Root cause analysis, i.e. error or fault diagnosis (in a hardware test environment G06F11/22; in a software test environment G06F11/36)} - G06F 3/0616: {in relation to life time, e.g. increasing Mean Time Between Failures [MTBF]} - G06F 3/0631: {by allocating resources to storage systems} - G06F 3/0653: {Monitoring storage devices or systems} - G06F 3/0689: {Disk arrays, e.g. RAID, JBOD} - G06F 11/0727: {in a storage system, e.g. in a DASD or network based storage system (drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)} - G06F 11/0751: {Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06)} - G06F 11/3034: {where the computing system component is a storage system, e.g. DASD based or network based (digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)} - G06F 11/3409: {for performance assessment} - G06F 11/3485: {for I/O devices} - G06F 11/3495: {for systems} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Embodiments of the present disclosure provide a method and apparatus for determining a cause of performance degradation of a storage system. The method comprises: monitoring performance of the storage system according to a predetermined policy; generating a performance degradation event from a result of the monitoring based on system performance baseline; in response to performance degradation event, obtaining information about the performance degradation; and offline analyzing the information to determine the cause of the performance degradation. Compared with the prior art, embodiments of the present disclosure can manage the system performance degradation effectively and continuously to minimize running costs and enable current performance profiling tools in the manner of insertion. ## Claims ### Claim 1 A method for determining a cause of performance degradation of a storage system, comprising: monitoring performance of the storage system according to a predetermined policy, wherein monitoring performance of the storage system according to a predetermined policy comprises tracking a performance metric of a monitoring target to obtain original performance data of the monitoring target; generating, based on a system performance baseline, a performance degradation event from a result of the monitoring, wherein generating the performance degradation event from the result of the monitoring based on the system performance baseline comprises comparing the original performance data with a latest system performance baseline and generating the performance degradation event based on a result of the comparing; in response to the performance degradation event, obtaining information about the performance degradation; analyzing the information offline to determine the cause of the performance degradation; and recording the performance degradation event as a four-element tuple, wherein the four-element tuple includes a latest performance statistic, an indicator of the performance degradation, a degree of the performance degradation, and a count for reaching the degree of the performance degradation. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the performance degradation event based on a result of the comparing comprises: generating the performance degradation event in response to determining that the original performance data is lower than the latest system performance baseline. ### Claim 3 (depends on 1) The method of claim 1, further comprising: detecting normal performance of the storage system to generate a detected system performance baseline; and generating the latest system performance baseline based on the detected system performance baseline. ### Claim 4 (depends on 3) The method of claim 3, wherein detecting normal performance of the storage system comprises: detecting the normal performance of the storage system periodically; or detecting the normal performance of the storage system in response to a configuration change of the storage system, wherein the configuration change comprises at least one of the following: hardware change, work load change, booting up of the storage system and mounting of a file system. ### Claim 5 (depends on 3) The method of claim 3, wherein generating the latest system performance baseline based on the detected system performance baseline comprises: generating the latest system performance baseline by weighting a current system performance baseline and the detected system performance baseline. ### Claim 6 (depends on 1) The method of claim 1, wherein obtaining information about the performance degradation in response to the performance degradation event comprises: performing a first action based on a degree of the performance degradation, wherein the first action comprises at least one of the following: analyzing, on a first target, a first layer; analyzing, on the first target, a plurality of layers in vertical; and analyzing, on the first layer, a plurality of targets in horizontal. ### Claim 7 (depends on 6) The method of claim 6, wherein obtaining information about the performance degradation in response to the performance degradation event further comprises: monitoring resource utilization of the storage system; performing a second action based on the resource utilization; and performing the first action to obtain the information about the performance degradation, wherein the second action comprises: ranking, based on the resource utilization, a plurality of targets that utilize a specific resource; and limiting the utilization of the specific resource by at least one of the plurality of targets with a rank higher than a predetermined threshold. ### Claim 8 (depends on 6) The method of claim 6, wherein the first action is performed by at least one of the following entities: a system built-in performance profiling tool, a plug-in performance profiling tool, and a third-party performance profiling tool. ### Claim 9 (depends on 1) The method of claim 1, wherein information related to the performance degradation comprises at least one of the following: related event information, system tracking information, latest performance statistics information, and relevant system configuration information. ### Claim 10 An apparatus for determining a cause of performance degradation of a storage system, including one or more processors and one or more memories configured to perform operations, comprising: monitoring performance of the storage system according to a predetermined policy, wherein monitoring performance of the storage system according to a predetermined policy comprises tracking a performance metric of a monitoring target to obtain original performance data of the monitoring target; generating, based on a system performance baseline, a performance degradation event from a result of the monitoring, wherein generating the performance degradation event from the result of the monitoring based on the system performance baseline comprises comparing the original performance data with a latest system performance baseline and generating the performance degradation event based on a result of the comparing; obtaining information about the performance degradation in response to the performance degradation event; analyzing the information offline to determine the cause of the performance degradation; and recording the performance degradation event as a four-element tuple, wherein the four-element tuple includes a latest performance statistic, an indicator of the performance degradation, a degree of the performance degradation, and a count for reaching the degree of the performance degradation. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein generating the performance degradation event based on a result of the comparing includes: generating the performance degradation event in response to determining that the original performance data is lower than the latest system performance baseline. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein the operations further comprise: detecting normal performance of the storage system to generate a detected system performance baseline; and generating the latest system performance baseline based on the detected system performance baseline. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the operations further comprise: detecting the normal performance of the storage system periodically; or detecting the normal performance of the storage system in response to a configuration change of the storage system, wherein the configuration change comprises at least one of the following: hardware change, work load change, booting up of the storage system and mounting of a file system. ### Claim 14 (depends on 12) The apparatus of claim 12, wherein the operations further comprise generating the latest system performance baseline by weighting a current system performance baseline and the detected system performance baseline. ### Claim 15 (depends on 10) The apparatus of claim 10, wherein the operations further comprise: performing a first action based on a degree of performance degradation, wherein the first action comprises at least one of the following: analyzing, on a first target, a first layer; analyzing, on the first target, a plurality of layers in vertical; and analyzing, on the first layer, a plurality of targets in horizontal. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the operations further comprise: monitoring resource utilization of the storage system; performing a second action based on the resource utilization; and performing the first action to obtain the information about the performance degradation, wherein the second action comprises at least one of the following: ranking, based on the resource utilization, a plurality of targets that utilize a specific resources; and limiting the utilization of the specific resources by at least one of the plurality of targets with a rank higher than a predetermined threshold.