# US 12,770,051 B2 · Systems and methods for machine learning interpretability

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,770,051 B2
- Document ID: US12770051B2
- Status: Granted
- Kind: B2
- Application number: 17/073,777
- Filed: 2020-10-19
- Published: 2021-04-22
- Granted: 2026-10-20
- Earliest claimed: 2019-10-19
- Assignees: Kinaxis Inc.
- Applicants: Kinaxis Inc.
- Inventors: Behrouz Haji Soleimani; Andrea Pagotto; Seyednaser Nourashrafeddin; Chantal Bisson-Krol
- Web page: https://patentgrep.com/patent/US12770051B2

## CPC

- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods and systems that provide machine learning interpretability. SHAP values of historical and predicted data, along with features of both, are used to provide a measure of the impact of training data points on a predictions. Removal of an individual training data point from a training data set, followed by comparing the resulting prediction with that obtained by the full training data set, also provides a measure of influence of individual training data points on forecasts.

## Claims

### Claim 1

A method comprising:
training, by a processor, a regression machine learning model using training data comprising training data points for a historical time-series;
utilizing, by the processor, the trained regression machine learning model to generate a prediction;
receiving, by a machine learning interpretability module, the training data, the trained model and the prediction;
determining, for each of the training data points of the training data, a measure of influence for each of the training data points, wherein a procedure for determining the measure of influence for each of the training data points comprises:
a) removing, by the machine learning interpretability module, the training data point from the training data to form an amended training data set;
b) retraining, by the machine learning interpretability module, the trained regression machine learning model on the amended training data set;
c) utilizing, by the machine learning interpretability module, the retrained regression machine learning model to generate an amended prediction;
d) determining, by the machine learning interpretability module, a difference between the prediction generated by the trained regression machine learning model and the amended prediction generated by the retrained regression machine learning model;
e) determining, by the machine learning interpretability module, a measure of influence for the removed training data point, based on the difference;
f) return the removed training data point to the amended training data set along with the measure of influence for the removed training data point;
g) remove, by the machine learning interpretability module, another training data point from the training data to form another amended training data set; and
repeat the procedure a)-g) for determining and returning the measure of influence for each of the training data points in the training data; and
visually outputting the measure of influence of each of the training data points to a graphical user interface so that a visualization of the training data comprises a visualization of the measure of influence of each of the training points on the prediction.

### Claim 2 (depends on 1)

The method of claim 1, wherein the visualization of the training data comprises a gradient representation of the measure of influence of each of the training points.

### Claim 3 (depends on 2)

The method of claim 2, wherein the gradient representation comprises a gradient key that indicates the measure of influence of each of the training data points.

### Claim 4 (depends on 1)

The method of claim 1, wherein the visualization of the measure of influence of each of the training points on the prediction comprises a visualization of changes from a resulting amended data forecast from the amended training data set for a selected training data point relative to a full data forecast from the training data.

### Claim 5 (depends on 4)

The method of claim 4, wherein the graphical user interface is configured to allow a user to select a given one of the training data points from the training data set and visualize the changes from the resulting amended data forecast from the amended training data set for the selected training data point relative to the full data forecast from the training data.

### Claim 6

A system comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the system to:
train, by a processor, a regression machine learning model using training data comprising training data points for a historical time-series;
utilize, by the processor, the trained regression machine learning model to generate a prediction;
receive, by a machine learning interpretability module, the training data, the trained model and the prediction;
determine, for each of the training data points of the training data, a measure of influence for the training data points, wherein a procedure for determining the measure of influence for each of the one or more training data points comprises:
a) remove, by the machine learning interpretability module, a first training data point from the training data to form an amended training data set;
b) retrain, by the machine learning interpretability module, the trained regression machine learning model on the amended training data set;
c) utilize, by the machine learning interpretability module, the retrained regression machine learning model to generate an amended prediction;
d) determine, by the machine learning interpretability module, a difference between the prediction generated by the trained regression machine learning model and the amended prediction generated by the retrained regression machine learning model;
e) determine, by the machine learning interpretability module, a measure of influence for the removed first training data point, based on the difference;
f) return the removed first training data point to the amended training data set along with the measure of influence for the removed training data point;
g) remove, by the machine learning interpretability module, another training data point from the training data to form another amended training data set; and
repeat the procedure a)-g) for determining and returning the measure of influence for each of the training data points in the training data; and
visually output the measure of influence to the one or more training data points to a graphical user interface so that a visualization of the training data comprises a visualization of the measure of influence of each of the training points on the prediction.

### Claim 7 (depends on 6)

The system of claim 6, further configured to provide the visualization of the training data comprising a gradient representation of the measure of influence of each of the training points.

### Claim 8 (depends on 7)

The system of claim 7, wherein the gradient representation comprises a gradient key that indicates the measure of influence of each of the training data points.

### Claim 9 (depends on 6)

The system of claim 6, wherein the visualization of the measure of influence of each of the training points on the prediction comprises a visualization of changes from a resulting amended data forecast from the amended training data set for a selected training data point relative to a full data forecast from the training data.

### Claim 10 (depends on 9)

The system of claim 9, wherein the graphical user interface is configured to allow a user to select a given one of the training data points from the training data set and visualize the changes from the resulting amended data forecast from the amended training data set for the selected training data point relative to the full data forecast from the training data.

### Claim 11

A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
train, by a processor, a regression machine learning model using training data comprising training data points for a historical time-series;
utilize, by the processor, the trained regression machine learning model to generate a prediction;
receive, by a machine learning interpretability module, the training data, the trained model and the prediction;
determine, for each of the training data points of the training data, a measure of influence for each of the training data points, wherein a procedure determining the measure of influence for each of the training data points comprises:
a) remove, by the machine learning interpretability module, the training data point from the training data to form an amended training data set;
b) retrain, by the machine learning interpretability module, the trained regression machine learning model on the amended training data set;
c) utilize, by the machine learning interpretability module, the retrained regression machine learning model to generate an amended prediction;
d) determine, by the machine learning interpretability module, a difference between the prediction generated by the trained regression machine learning model and the amended prediction generated by the retrained regression machine learning model;
e) determine, by the machine learning interpretability module, a measure of influence for the removed training data point, based on the difference;
f) return the removed training data point to the amended training data set along with the measure of influence for the removed training data point; and
g) remove, by the machine learning interpretability module, another training data point from the training data to form another amended training data set; and
repeat the procedure a)-g) for determining and returning the measure of influence for each of the training data points in the training data; and
visually output the measure of influence of the one or more training data points to a graphical user interface so that a visualization of the training data comprises a visualization of the measure of influence of each of the training points on the prediction.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein instructions that when executed by a computer, further cause the computer to provide the visualization of the training data comprising a gradient representation of the measure of influence of each of the training points.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein the gradient representation comprises a gradient key that indicates the measure of influence of each of the one or more of the training data points.

### Claim 14 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the visualization of the of the measure of influence of each of the training points on the prediction comprises a visualization of changes from a resulting amended data forecast from the amended training data set for a selected training data point relative to a full data forecast from the training data.

### Claim 15 (depends on 14)

The non-transitory computer-readable storage medium of claim 14, wherein the graphical user interface is configured to allow a user to select a given one of the training data points from the training data set and visualize the changes from the resulting amended data forecast from the amended training data set for the selected training data point relative to the full data forecast from the training data.

### Claim 16 (depends on 1)

The method of claim 1, wherein the training data points for the historical time-series removed by the machine learning interpretability module to form amended training data sets are non-synthetic data points from raw data for real events.

### Claim 17 (depends on 6)

The system of claim 6, wherein the training data points for the historical time-series removed by the machine learning interpretability module to form amended training data sets are non-synthetic data points from raw data for real events.

### Claim 18 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the training data points for the historical time-series removed by the machine learning interpretability module to form amended training data sets are non-synthetic data points from raw data for real events.