# US 2021/0241130 A1 · Performance Improvement Recommendations for Machine Learning Models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241130 A1 - Document ID: US20210241130A1 - Status: Published application - Kind: A1 - Application number: 16/778,084 - Filed: 2020-01-31 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: RSA Security Inc. - Applicants: RSA Security LLC - Inventors: Alex Zaslavsky; Arkady Koganov; Anatoly Gendelev - Web page: https://patentgrep.com/patent/US20210241130A1 ## CPC - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Techniques are provided for generating performance improvement recommendations for machine learning models. One method comprises evaluating performance metrics for multiple implementations of a machine learning model; computing a performance score that aggregates the performance metrics for a given machine learning model implementation; and recommending a modification to the given machine learning model implementation based on the performance score by evaluating one or more performance metrics for the given implementation relative to at least one additional performance metric for the given implementation, wherein the recommended modification is based on a performance with the recommended modification for another implementation. A given performance metric may be weighted based on an expected improvement from modifying a factor related to the given performance metric. The recommended modification to the given machine learning model implementation may comprise an indication of the expected improvement for the modification. ## Claims ### Claim 1 A method comprising: evaluating a plurality of performance metrics for each of a plurality of implementations of a machine learning model; computing a performance score that aggregates the plurality of performance metrics for a given implementation of the machine learning model; and recommending at least one modification to the given implementation of the machine learning model based at least in part on the performance score by evaluating one or more of the plurality of performance metrics for the given implementation of the machine learning model relative to one or more additional ones of the plurality of performance metrics for the given implementation of the machine learning model, wherein the at least one recommended modification is based at least in part on a performance with the at least one recommended modification for at least one other implementation, wherein the method is performed by at least one processing device comprising a processor coupled to a memory. ### Claim 2 (depends on 1) The method of claim 1, wherein the performance metrics are grouped by category and a different performance score is computed for each category. ### Claim 3 (depends on 2) The method of claim 2, wherein the performance score is computed for a given category by summing the performance metrics for the given category. ### Claim 4 (depends on 3) The method of claim 3, further comprising providing the performance score for each category in a sorted list. ### Claim 5 (depends on 1) The method of claim 1, wherein each of the plurality of performance metrics are weighted based, at least in part, on an expected improvement for a modification of a factor related to each performance metric. ### Claim 6 (depends on 5) The method of claim 5, wherein the expected improvement for a first performance metric is based at least in part on a performance of the first performance metric for at least one other implementation. ### Claim 7 (depends on 5) The method of claim 5, wherein different weightings are obtained for multiple groupings of implementations having similar characteristics based on one or more predefined similarity criteria. ### Claim 8 (depends on 1) The method of claim 1, wherein the recommending the at least one modification to the given implementation of the machine learning model further comprises providing an indication of an expected improvement for the one or more modifications, wherein the expected improvement is based at least in part on a performance with the one or more modifications for at least one other implementation. ### Claim 9 (depends on 1) The method of claim 1, wherein the evaluating the plurality of performance metrics for the given implementation of the machine learning model relative to one or more additional ones of the plurality of performance metrics for the given implementation of the machine learning model further comprises determining one or more reasons for a lower score. ### Claim 10 An apparatus comprising: at least one processing device comprising a processor coupled to a memory; the at least one processing device being configured to implement the following steps: evaluating a plurality of performance metrics for each of a plurality of implementations of a machine learning model; computing a performance score that aggregates the plurality of performance metrics for a given implementation of the machine learning model; and recommending at least one modification to the given implementation of the machine learning model based at least in part on the performance score by evaluating one or more of the plurality of performance metrics for the given implementation of the machine learning model relative to one or more additional ones of the plurality of performance metrics for the given implementation of the machine learning model, wherein the at least one recommended modification is based at least in part on a performance with the at least one recommended modification for at least one other implementation. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein the performance metrics are grouped by category and a different performance score is computed for each category, and wherein the performance score is computed for a given category by summing the performance metrics for the given category. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein each of the plurality of performance metrics are weighted based, at least in part, on an expected improvement for a modification of a factor related to each performance metric, and wherein the expected improvement for a first performance metric is based at least in part on a performance of the first performance metric for at least one other implementation. ### Claim 13 (depends on 10) The apparatus of claim 10, wherein the at least one modification to the given implementation of the machine learning model further comprises providing an indication of an expected improvement for the one or more modifications, and wherein the expected improvement is based at least in part on a performance with the one or more modifications for at least one other implementation. ### Claim 14 (depends on 10) The apparatus of claim 10, wherein the evaluating the plurality of performance metrics for the given implementation of the machine learning model relative to one or more additional ones of the plurality of performance metrics for the given implementation of the machine learning model further comprises determining one or more reasons for a lower score. ### Claim 15 A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps: evaluating a plurality of performance metrics for each of a plurality of implementations of a machine learning model; computing a performance score that aggregates the plurality of performance metrics for a given implementation of the machine learning model; and recommending at least one modification to the given implementation of the machine learning model based at least in part on the performance score by evaluating one or more of the plurality of performance metrics for the given implementation of the machine learning model relative to one or more additional ones of the plurality of performance metrics for the given implementation of the machine learning model, wherein the at least one recommended modification is based at least in part on a performance with the at least one recommended modification for at least one other implementation. ### Claim 16 (depends on 15) The non-transitory processor-readable storage medium of claim 15, wherein the performance metrics are grouped by category and a different performance score is computed for each category, and wherein the performance score is computed for a given category by summing the performance metrics for the given category. ### Claim 17 (depends on 15) The non-transitory processor-readable storage medium of claim 15, wherein each of the plurality of performance metrics are weighted based, at least in part, on an expected improvement for a modification of a factor related to each performance metric, and wherein the expected improvement for a first performance metric is based at least in part on a performance of the first performance metric for at least one other implementation. ### Claim 18 (depends on 15) The non-transitory processor-readable storage medium of claim 15, wherein the at least one modification to the given implementation of the machine learning model further comprises providing an indication of an expected improvement for the one or more modifications. ### Claim 19 (depends on 18) The non-transitory processor-readable storage medium of claim 18, wherein the expected improvement is based at least in part on a performance with the one or more modifications for at least one other implementation. ### Claim 20 (depends on 15) The non-transitory processor-readable storage medium of claim 15, wherein the evaluating the plurality of performance metrics for the given implementation of the machine learning model relative to one or more additional ones of the plurality of performance metrics for the given implementation of the machine learning model further comprises determining one or more reasons for a lower score.