# US 2025/0094881 A1 · Machine learning framework for generating multiple machine-learned models based on different sets of training data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094881 A1
- Document ID: US20250094881A1
- Status: Published application
- Kind: A1
- Application number: 18/368,525
- Filed: 2023-09-14
- Published: 2025-03-20
- Earliest claimed: 2022-12-05
- Assignees: Predict HQ Limited
- Applicants: Predict Hq Limited
- Inventors: Xuxu Wang; Xiping Fu
- Web page: https://patentgrep.com/patent/US20250094881A1

## CPC

- G06N 20/20: Ensemble learning
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A framework for generating multiple machine-learned (ML) models in order to learn the impact of real-world events is provided. In one technique, sets of feature values (FVs) are stored, each FV set corresponding to a different feature. Also, sets of artificial FVs (AFVs) are generated. For each generated AFV set: (1) a training dataset is generated based on that AFV set and the multiple FV sets; (2) a model is generated based on the training dataset; (3) a ranking of the multiple features is generated based on the model; and (4) the ranking is stored in a dictionary corresponding to the generated AFV set. For each feature, a rank pair of the feature and an AF is determined from each dictionary. Based on a set of rank pairs associated with the feature, it is determined whether there is significant correlation between the feature and a response variable of the models.

## Claims

### Claim 1

A method comprising:
generating a plurality of sets of artificial feature values, wherein generating the plurality of sets of artificial feature values comprises:
identifying a set of feature values in a plurality of sets of feature values;
shuffling the set of feature values to generate a set of artificial feature values, wherein the set of artificial feature values is included in the plurality of sets of artificial feature values;
for each set of artificial feature values of the plurality of sets of artificial feature values:
generating a training data set based on said each set of artificial feature values and the plurality of sets of feature values;
wherein each set of feature values in the plurality of sets of feature values corresponds to a different feature of a plurality of features;
generating a machine-learned model based on the training dataset;
based on the machine-learned model, generating a ranking of the plurality of features;
adding the machine-learned model to a set of machine-learned models;
storing the ranking in a dictionary, of a plurality of dictionaries, that corresponds to said each set of artificial feature values;
for each feature of the plurality of features:
for each dictionary of the plurality of dictionaries:
generating a rank pair comprising (1) a rank of said each feature, in said each dictionary, relative to other features of the plurality of features and (2) a rank of an artificial feature, in said each dictionary, relative to other features of the plurality of features;
adding the rank pair to a set of rank pairs associated with said each feature;
wherein after generating the rank pair for each dictionary of the plurality of dictionaries, the set of rank pairs is a plurality of rank pairs associated with said each feature;
based on the plurality of rank pairs associated with said each feature, determining whether there is significant correlation between said each feature and a response variable of the set of machine-learned models;
wherein the method is performed by one or more computing devices.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining whether there is significant correlation comprises:
conducting a statistical test based on the set of rank pairs that results in a p-value for said each feature;
determining whether the p-value is less than a particular threshold.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the plurality of sets of artificial feature values comprises generating a particular set of artificial feature values, in the plurality of sets of artificial feature values, based on a particular set of feature values in the plurality of sets of feature values.

### Claim 4 (depends on 3)

The method of claim 3, wherein generating the particular set of artificial feature values comprises randomly shuffling feature values in the particular set of feature values to generate the particular set of artificial feature values.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the plurality of sets of artificial feature values comprises, for each set of feature values in the plurality of sets of feature values, generating multiple sets of artificial feature values.

### Claim 6 (depends on 1)

The method of claim 1, wherein generating the machine-learned model comprises using a gradient boosting machine to generate a regression model.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
based on the machine-learned model, generating a feature importance score for each feature in the plurality of features;
wherein generating the ranking of the plurality of features is based on the feature importance score for each feature in the plurality of features.

### Claim 8 (depends on 7)

The method of claim 7, wherein generating the feature importance score is performed using a permutation feature importance technique.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the training data set comprises:
receiving remainder data that is based on a demand data set that comprises a particular time series data set that reflects user behavior over time; and
including, in each training instance of the training data set, for the response variable, a value that is from the remainder data.

### Claim 10 (depends on 9)

The method of claim 9, wherein the remainder data is time series data that comprises a plurality of values, each of which corresponds to a different time period of a plurality of time periods.

### Claim 11

One or more storage media storing instructions which, when executed by one or more computing devices, cause:
generating a plurality of sets of artificial feature values, wherein generating the plurality of sets of artificial feature values comprises:
identifying a set of feature values in a plurality of sets of feature values;
shuffling the set of feature values to generate a set of artificial feature values, wherein the set of artificial feature values is included in the plurality of sets of artificial feature values;
for each set of artificial feature values of the plurality of sets of artificial feature values:
generating a training data set based on said each set of artificial feature values and the plurality of sets of feature values;
wherein each set of feature values in the plurality of sets of feature values corresponds to a different feature of a plurality of features;
generating a machine-learned model based on the training dataset;
based on the machine-learned model, generating a ranking of the plurality of features;
adding the machine-learned model to a set of machine-learned models;
storing the ranking in a dictionary, of a plurality of dictionaries, that corresponds to said each set of artificial feature values;
for each feature of the plurality of features:
for each dictionary of the plurality of dictionaries:
generating a rank pair comprising (1) a rank of said each feature, in said each dictionary, relative to other features of the plurality of features and (2) a rank of an artificial feature, in said each dictionary, relative to other features of the plurality of features;
adding the rank pair to a set of rank pairs associated with said each feature;
wherein after generating the rank pair for each dictionary of the plurality of dictionaries, the set of rank pairs is a plurality of rank pairs associated with said each feature;
based on the plurality of rank pairs associated with said each feature, determining whether there is significant correlation between said each feature and a response variable of the set of machine-learned models.

### Claim 12 (depends on 11)

The one or more storage media of claim 11, wherein determining whether there is significant correlation comprises:
conducting a statistical test based on the set of rank pairs that results in a p-value for said each feature;
determining whether the p-value is less than a particular threshold.

### Claim 13 (depends on 11)

The one or more storage media of claim 11, wherein generating the plurality of sets of artificial feature values comprises generating a particular set of artificial feature values, in the plurality of sets of artificial feature values, based on a particular set of feature values in the plurality of sets of feature values.

### Claim 14 (depends on 13)

The one or more storage media of claim 13, wherein generating the particular set of artificial feature values comprises randomly shuffling feature values in the particular set of feature values to generate the particular set of artificial feature values.

### Claim 15 (depends on 11)

The one or more storage media of claim 11, wherein generating the plurality of sets of artificial feature values comprises, for each set of feature values in the plurality of sets of feature values, generating multiple sets of artificial feature values.

### Claim 16 (depends on 11)

The one or more storage media of claim 11, wherein generating the machine-learned model comprises using a gradient boosting machine to generate a regression model.

### Claim 17 (depends on 11)

The one or more storage media of claim 11, wherein the instructions, when executed by the one or more processors, further cause:
based on the machine-learned model, generating a feature importance score for each feature in the plurality of features;
wherein generating the ranking of the plurality of features is based on the feature importance score for each feature in the plurality of features.

### Claim 18 (depends on 17)

The one or more storage media of claim 17, wherein generating the feature importance score is performed using a permutation feature importance technique.

### Claim 19 (depends on 11)

The one or more storage media of claim 11, wherein generating the training data set comprises:
receiving remainder data that is based on a demand data set; and
including, in each training instance of the training data set, for the response variable, a value that is from the remainder data.

### Claim 20 (depends on 19)

The one or more storage media of claim 19, wherein the remainder data is time series data that comprises a plurality of values, each of which corresponds to a different time period of a plurality of time periods.
