# US 11,443,207 B2 · Aggregated feature importance for finding influential business metrics

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,207 B2
- Document ID: US11443207B2
- Status: Granted
- Kind: B2
- Application number: 16/816,400
- Filed: 2020-03-12
- Published: 2021-09-16
- Granted: 2022-09-13
- Earliest claimed: 2020-03-12
- Assignees: Capital One Services, LLC
- Applicants: Capital One Services, LLC
- Inventors: Yasong Zhou; Wayne Decesaris; Esmat Zare
- Web page: https://patentgrep.com/patent/US11443207B2

## CPC

- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G06Q 10/0639: Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q 10/10: Office automation; Time management
- G06Q 30/016: After-sales
- G06Q 30/0203: Market surveys; Market polls
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Systems, methods, and apparatuses for determining feature importance of analytics data in predicting a response value include receiving data records, each data record including a response value and values of features associated with the response value; splitting the data records into datasets, each dataset including a part of the data records; generating a machine learning model using each of the datasets, the machine learning model outputting one or more predicting features having influence in predicting the response value for each of the datasets; determining an important feature based on the one or more predicting features; and generating report data indicating that a business metric associated with the important feature has the highest predicted influence among the features on predicting the response value.

## Claims

### Claim 1

A system for determining feature importance of analytics data in predicting a response value, the system comprising:
at least one processor; and
at least one non-transitory memory storing instructions to perform operations when executed by the at least one processor including:
receiving data records, each data record comprising a response value and values of features associated with the response value;
splitting the data records into datasets, each dataset comprising a part of the data records;
generating a machine learning model using each of the datasets, the machine learning model outputting one or more predicting features having influence in predicting the response value for each of the datasets;
determining an important feature based on the one or more predicting features; and
generating report data indicating that a business metric associated with the important feature has the highest predicted influence among the features on predicting the response value.

### Claim 2 (depends on 1)

The system of claim 1, wherein the data records are received from a customer data collector.

### Claim 3 (depends on 2)

The system of claim 2, wherein the data records comprise survey data records.

### Claim 4 (depends on 2)

The system of claim 2, wherein the values of the features comprise a metric value representing an operational performance of the customer data collector.

### Claim 5 (depends on 2)

The system of claim 2, wherein the response value represents a satisfaction level directed to the customer data collector.

### Claim 6 (depends on 1)

The system of claim 1, wherein the operations further comprise:
determining random numbers, each random number being associated with one of the datasets; and
generating the machine learning model using the each of the datasets and one of the random numbers.

### Claim 7 (depends on 6)

The system of claim 6, wherein the generating the machine learning model comprises initializing the machine learning model using the one of the random numbers.

### Claim 8 (depends on 1)

The system of claim 1, wherein the machine learning model comprises one of a gradient boosting machine model, a decision tree model, or a random forest model.

### Claim 9 (depends on 1)

The system of claim 1, wherein each of the datasets comprises a training dataset for training the machine learning model and a validation dataset for validating the machine learning model.

### Claim 10 (depends on 1)

The system of claim 1, wherein the operations further comprise:
determining feature importance values using the machine learning model, wherein each feature importance value represents a contribution of influence level of a respective one of the features has in predicting the response value; and
determining the one or more predicting features based on the feature importance values.

### Claim 11 (depends on 10)

The system of claim 10, wherein the one or more predicting features reflect features having a predetermined number of highest contributions of influence level among the feature importance values.

### Claim 12 (depends on 1)

The system of claim 1, wherein determining the important feature comprises:
determining a frequency of each of the one or more predicting features; and
determining the important feature based on a predicting feature associated with a highest frequency.

### Claim 13 (depends on 1)

The system of claim 1, wherein operations further comprise:
determining an area under receiver operating characteristics (AUROC) for the machine learning model; and
determining a confidence interval for the AUROC.

### Claim 14

A computer-implemented method for determining feature importance of analytics data in predicting a response value, the method comprising:
receiving data records, each data record comprising a customer satisfaction score and values of features associated with the response value;
splitting the data records into datasets, each dataset comprising a training dataset for training a machine learning model and a validation dataset for validating the machine learning model;
generating the machine learning model using each of the datasets, the machine learning model outputting one or more predicting features having influence in predicting the response value for each of the datasets;
determining an important feature based on the one or more predicting features; and
generating report data indicating that a business metric associated with the important feature has the highest predicted influence among the features on predicting the response value.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
determining random numbers, each random number being associated with one of the datasets; and
generating the machine learning model based on the datasets and one of the random numbers.

### Claim 16 (depends on 15)

The method of claim 15, wherein generating the machine learning model further comprises initializing the machine learning model using one of the random numbers.

### Claim 17 (depends on 14)

The method of claim 14, wherein the machine learning model comprises one of a gradient boosting machine model, a decision tree model, or a random forest model.

### Claim 18 (depends on 14)

The method of claim 14, further comprising:
determining feature importance values using the machine learning model, wherein each feature importance value represents a contribution of influence level of a respective one of the features has in predicting the response value; and
determining the one or more predicting features as features having feature importance values representing a predetermined number of highest contributions of the influence level.

### Claim 19 (depends on 14)

The method of claim 14, further comprising:
determining a frequency of each of the one or more predicting features; and
determining the important feature based on a predicting feature associated with a highest frequency.

### Claim 20

A non-transitory computer-readable storage medium comprising instructions for determining feature importance of analytics data in predicting a response value, wherein instructions when executed by a processor perform operations comprising:
receiving survey response data records, each survey response data record comprising a response value and values of features associated with the response value;
splitting the data records into datasets, each dataset comprising a part of the data records;
generating a machine learning model using each of the datasets, the machine learning model outputting one or more predicting features having influence in predicting the response value for each of the datasets;
determining an important feature based on the one or more predicting features; and
generating report data indicating that a business metric associated with the important feature has the highest predicted influence among the features on predicting the response value.