# US 12,008,484 B2 · Aggregated feature importance for finding influential business metrics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,484 B2 - Document ID: US12008484B2 - Status: Granted - Kind: B2 - Application number: 17/881,273 - Filed: 2022-08-04 - Published: 2022-11-24 - Granted: 2024-06-11 - Earliest claimed: 2020-03-12 - Assignees: Capital One Services, LLC - Applicants: Capital One Services, LLC - Inventors: Yasong Zhou; Wayne Decesaris; Esmat Zare - Web page: https://patentgrep.com/patent/US12008484B2 ## CPC - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06N 20/20: Ensemble learning - G06Q 10/0639: Performance analysis of employees; Performance analysis of enterprise or organisation operations - G06Q 10/10: Office automation; Time management - G06Q 30/016: After-sales - G06Q 30/0203: Market surveys; Market polls - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract Systems, methods, and apparatuses for determining feature importance of analytics data in predicting a response value include receiving data records, each data record including a response value and values of features associated with the response value; splitting the data records into datasets, each dataset including a part of the data records; generating a machine learning model using each of the datasets, the machine learning model outputting one or more predicting features having influence in predicting the response value for each of the datasets; determining an important feature based on the one or more predicting features; and generating report data indicating that a business metric associated with the important feature has the highest predicted influence among the features on predicting the response value. ## Claims ### Claim 1 A computer-implemented method for determining feature importance of analytics data in predicting a response value, the method comprising: splitting data records into datasets, each dataset comprising a training dataset for training a machine learning model; generating the machine learning model using each of the datasets, the machine learning model outputting one or more predicting features having influence in predicting the response value for each of the datasets; determining an important feature based on the one or more predicting features; and generating report data indicating that a business metric associated with the important feature has the highest predicted influence among the features on predicting the response value. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: determining random numbers, each random number being associated with one of the datasets; and generating the machine learning model based on the datasets and one of the random numbers. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein generating the machine learning model further comprises initializing the machine learning model using one of the random numbers. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein each data record comprises a customer satisfaction score and values of features associated with the response value. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein each dataset further comprises a validation dataset for validating the machine learning model. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein: training the machine learning model is completed upon correct validation using the validation dataset, and correct validation comprising correctly outputting, by the machine learning model, one or more target variables after receiving the validation dataset as input. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: determining feature importance values using the machine learning model, wherein each feature importance value represents a contribution of influence level of a respective one of the features has in predicting the response value; and determining the one or more predicting features as features having feature importance values representing a predetermined number of highest contributions of the influence level. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein determining the important feature comprises: determining a frequency of each of the one or more predicting features, and determining the important feature based on a predicting feature associated with a highest frequency. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising: determining an area under receiver operating characteristics (AUROC) for the machine learning model; and determining a confidence interval for the AUROC. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, further comprising: estimating a distribution of AUROC values; and analyzing the distribution to determine whether an accuracy criterion for the machine learning model has been satisfied. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, wherein analyzing the distribution comprises: determining an average AUROC value, and comparing the average AUROC value to a predetermined threshold. ### Claim 12 (depends on 10) The computer-implemented method of claim 10, wherein analyzing the distribution further comprises determining a confidence interval for the AUROC based on the comparison. ### Claim 13 (depends on 10) The computer-implemented method of claim 10, wherein the accuracy criterion is dependent upon at least one selected from the group of the standard deviation of the distribution, the mean of the distribution, the median of the distribution, the mode of the distribution, and another statistic of the distribution. ### Claim 14 A system, comprising: at least one processor; and at least one non-transitory memory storing instructions to perform operations when executed by the at least one processor including: splitting data records into datasets, each dataset comprising a training dataset for training a machine learning model; generating the machine learning model using each of the datasets, the machine learning model outputting one or more predicting features having influence in predicting the response value for each of the datasets; determining an important feature based on the one or more predicting features; and generating report data indicating that a business metric associated with the important feature has the highest predicted influence among the features on predicting the response value. ### Claim 15 (depends on 14) The system of claim 14, wherein the one or more predicting features comprises at least one selected from the group of one or more predictors that concern a transaction and one or more predictors that concern future user behavior. ### Claim 16 (depends on 15) The system of claim 15, wherein the one or more predictors that concern the transaction comprise at least one selected from the group of an ease of the transaction, a behavior of a counterparty to the transaction, a term of the transaction, a perceived benefit of the transaction, and a relative value of the transaction compared to one or more other potential transactions. ### Claim 17 (depends on 15) The system of claim 15, wherein the one or more predictors that concern future user behavior comprise at least one selected from the group of a likelihood that the user will repeat the transaction, a likelihood that the user will recommend the transaction to others, and a reason of unsatisfaction. ### Claim 18 A non-transitory computer-readable storage medium comprising instructions for determining feature importance of analytics data in predicting a response value, wherein instructions when executed by a processor perform operations comprising: splitting data records into datasets, each dataset comprising a training dataset for training a machine learning model; generating the machine learning model using each of the datasets, the machine learning model outputting one or more predicting features having influence in predicting the response value for each of the datasets; determining an important feature based on the one or more predicting features; and generating report data indicating that a business metric associated with the important feature has the highest predicted influence among the features on predicting the response value. ### Claim 19 (depends on 18) The non-transitory computer-readable storage medium of claim 18, further comprising: determining feature importance values using the machine learning model, wherein each feature importance value represents a contribution of influence level of a respective one of the features has in predicting the response value; and determining the one or more predicting features as features having feature importance values representing a predetermined number of highest contributions of the influence level. ### Claim 20 (depends on 18) The non-transitory computer-readable storage medium of claim 18, wherein determining the important feature comprises: determining a frequency of each of the one or more predicting features, and determining the important feature based on a predicting feature associated with a highest frequency.