# US 10,872,025 B1 · Automatic performance testing and performance regression analysis in a continuous integration environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,025 B1 - Document ID: US10872025B1 - Status: Granted - Kind: B1 - Application number: 16/237,146 - Filed: 2018-12-31 - Granted: 2020-12-22 - Earliest claimed: 2018-12-31 - Assignees: The MathWorks, Inc. - Applicants: The MathWorks, Inc. - Inventors: Meng-Ju Wu; Seihyung Lee; Ganesh Krishnamurthy; Murali Gopinathan; Robert O. Aberg - Web page: https://patentgrep.com/patent/US10872025B1 ## CPC - G06F 11/3466: {Performance evaluation by tracing or monitoring} - G06F 8/34: Graphical or visual programming - G06F 8/75: Structural analysis for program understanding - G06F 11/3409: {for performance assessment} - G06F 11/3419: {by assessing time} - G06F 11/3612: {by runtime analysis (performance monitoring G06F11/3466)} - G06F 11/3636: {by tracing the execution of the program} - G06F 11/3672: {Test management} - G06F 11/3696: {Methods or tools to render software testable} - G06F 11/3698: Environments for analysis, debugging or testing of software - G06F 11/3414: {Workload generation, e.g. scripts, playback} - G06F 2201/81: Threshold - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A performance testing and analysis environment performs tests on programming code and identifies any performance regressions. Tests exhibiting performance regressions are rerun, and the initial regressions are confirmed as genuine regressions or not. Analysis of the code under test and the performance under the tests are analyzed. The root causes of the performance regressions are identified. The process is fully automated and is performed quickly. ## Claims ### Claim 1 A method performed by a processor of a computing device, comprising: recording performance information during performance tests performed on programming code; identifying a performance regression that occurred in multiple of the performance tests, the identifying comprising: analyzing the recorded performance information to identify a performance change reflected in one or more performance metrics that occurred at in the multiple performance tests, and determining whether the performance regression is observable in that the performance change in at least one of the one or more performance metrics for the multiple performance tests surpasses a tolerance level; analyzing the identified observable performance regression to generate a data structure for callable units of functionality of the programming code executed during performing the performance tests, the data structure containing call relationships among the callable units of functionality of the programming code and performance information of the callable units of functionality of the program code; for each of at least two callable units of functionality, analyzing the data structure to identify a cause of the identified performance regression; and outputting information about the identified cause on an output device. ### Claim 2 (depends on 1) The method of claim 1, wherein at least one run of the running of the performance test is run on a different platform than a platform on which the performance test has been run. ### Claim 3 (depends on 1) The method of claim 1, wherein the output information includes an identification of at least one change set for the identified portion of the programming code that likely is the identified cause of the identified performance regression. ### Claim 4 (depends on 1) The method of claim 1, wherein the performance test measures at least one of the performance metrics of memory usage, processor time, number of calls, throughput, network traffic, network time, disk time or number of input/output operations. ### Claim 5 (depends on 1) The method of claim 1, wherein the programming code is for one of a graphical model or a textual model. ### Claim 6 (depends on 5) The method of claim 5, wherein the programming code is for a graphical model, the cause of the performance regression includes a component in the programming code and wherein the method further comprises identifying a change in the graphical model that caused the generation of the performance regression includes a component in the programming code and wherein the method further comprises identifying a change in the graphical model that caused the generation of the component. ### Claim 7 (depends on 1) The method of claim 1, wherein the determining that the performance regression is observable comprises rerunning the one or more performance tests to determine that the performance change is reproducible. ### Claim 8 (depends on 1) The method of claim 1, wherein the tolerance level is configurable. ### Claim 9 (depends on 1) The method of claim 1, wherein the outputting information comprises generating a report regarding the identified cause. ### Claim 10 (depends on 9) The method of claim 9, further comprising determining an owner of the at least one callable unit functionality that is the identified cause and sending the report to the owner. ### Claim 11 (depends on 1) The method of claim 1, wherein the identified cause of the identified performance regression is at least one callable unit of functionality in the programming code. ### Claim 12 (depends on 11) The method of claim 11, wherein the identified at least one callable unit of functionality is at least one of a function, a procedure, a subroutine, a method or an element of a model. ### Claim 13 (depends on 1) The method of claim 1, wherein the programming code is generated from a graphical model for a target system, the performance tests are run on the target system and the cause of the performance regression includes a component in the programming code and wherein the method further comprises identifying a change in the graphical model that caused the generation of the component. ### Claim 14 A method performed by a processor of a computing device, comprising: recording performance information during performance tests performed on programming code; identifying a performance regression that occurred multiple of the performance tests, the identifying comprising: analyzing the recorded performance information to identify a performance change reflected in one or more performance metrics that occurred in the multiple performance tests, and determining whether the performance regression is observable in that the performance change in at least one of the one or more metrics for the multiple performance tests surpasses a tolerance level; analyzing the identified observable performance regression to generate a call graph for the callable units of functionality of the programming code executed during the performance tests, the call graph containing call relationships among the callable units of functionality of the programming code and performance information of the callable units of functionality of the program code; processing the call graph by applying at least one anchor point to the call graph, for each of at least two callable units of functionality, analyzing the call graph to identify a cause of the identified performance regression; and outputting information about the identified cause on an output device. ### Claim 15 A non-transitory computer-readable storage medium storing instructions that cause a processor of a computing device to perform the following: recording performance information during performance tests performed in programming code; identifying a performance regression that occurred at multiple of the performance tests, the identifying comprising: analyzing recorded performance information to identify a performance change reflected in one or more performance metrics that occurred in the multiple performance tests, and determining whether the performance regression is observable in the performance change in at least one of the one or more performance metrics for the multiple performance tests surpasses a tolerance level; analyzing the identified observable performance regression to generate a data structure for callable units of functionality of the programming code executed during performing the performance tests, the data structure containing call relationships among the callable units of functionality of the program code and performance information of the callable units of functionality of the programming code; for reach of at least two callable units of functionality, analyzing the data structure to identify a cause of the identified performance regression; and outputting information about the identified cause on an output device. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein at least one run of the rerun performance test is run on a different platform than a platform on which the given run of the performance test has been run. ### Claim 17 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the output information includes an identification of at least one change set for the identified portion of the programming code that likely is the identified cause of the identified performance regression. ### Claim 18 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the performance test measures at least one of the performance metrics of memory usage, processor time, number of calls, throughput, network traffic, network time, disk time or number of input/output operations. ### Claim 19 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the programming code is for one of a graphical model or a textual model. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein the programming code is for a graphical model, the cause of the performance regression includes a component in the programming code and wherein the medium further stores instructions for identifying a change in the graphical model that caused the generation of the performance regression includes a component in the programming code and for identifying a change in the graphical model that caused the generation of the component. ### Claim 21 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the determining that the performance regression is observable comprises rerunning the one or more performance tests to determine that the performance change is reproducible. ### Claim 22 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the tolerance level is configurable. ### Claim 23 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the outputting information comprises generating a report regarding the identified cause. ### Claim 24 (depends on 23) The non-transitory computer-readable storage medium of claim 23, further comprising determining an owner of the at least one callable unit functionality that is the identified cause and sending the report to the owner. ### Claim 25 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the identified cause of the identified performance regression is at least one callable unit of functionality in the programming code. ### Claim 26 (depends on 25) The non-transitory computer-readable storage medium of claim 25, wherein the identified at least one callable unit of functionality is at least one of a function, a procedure, a subroutine, a method or an element of a model. ### Claim 27 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the programming code is generated from a graphical model for a target system, the performance tests are run on the target system and the cause of the performance regression includes a component in the programming code and wherein the storage medium further storage instructions for identifying a change in the graphical model that caused the generation of the component. ### Claim 28 A non-transitory computer-readable storage medium storing instructions that cause a processor of a computing device to perform the following: recording performance information during performance tests performed on programming code; identifying a performance regression that occurred multiple of the performance tests, the identifying comprising: analyzing the recorded performance information to identify a performance change reflected in one or more performance metrics that occurred in the multiple performance tests, and determining whether the performance regression is observable in that the performance change in at least one of the one or more metrics for the multiple performance tests surpasses a tolerance level; analyzing the identified observable performance regression to generate a call graph for the callable units of functionality of the programming code executed during the performance tests, the call graph containing call relationships among the callable units of functionality of the programming code and performance information of the callable units of functionality of the program code; processing the call graph by applying at least one anchor point to the call graph; for each of at least two callable units of functionality, analyzing the call graph to identify a cause of the identified performance regression; and outputting information about the identified cause on an output device. ### Claim 29 A computing device, comprising: a memory storing processor-executable instructions; a processor for executing the processor-executable instructions to perform the following: recording performance information during performance tests performed on programming code; identifying a performance regression that occurred in multiple of the performance tests, the identifying comprising: analyzing the recorded performance information to identify a performance change reflected in one or more performance metric that occurred in the multiple performance tests, and determining whether the performance regression is observable in that the performance change in at least one of the one or more performance metrics for the multiple performance tests surpasses a tolerance level; analyzing the identified observable performance regression to generate a data structure for callable units of functionality of the programming code executed during performing the performance tests, the data structure containing call relationships among the callable units of functionality of the programming code and performance information of the callable units of functionality of the program code; for each of at least two callable units of functionality, analyzing the data structure to identify a cause of the identified performance regression; and outputting information about the identified cause on an output device.