# US 12,183,465 B2 · Dynamic and locally-faithful explanation of machine learning models for precision medicine

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,465 B2
- Document ID: US12183465B2
- Status: Granted
- Kind: B2
- Application number: 18/135,851
- Filed: 2023-04-18
- Published: 2023-08-10
- Granted: 2024-12-31
- Earliest claimed: 2019-07-08
- Assignees: Koninklijke Philips N.V.
- Applicants: Koninklijke Philips N.V.
- Inventors: Gajendra Jung Katuwal; Bryan Conroy; Jonathan Rubin
- Web page: https://patentgrep.com/patent/US12183465B2

## CPC

- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/00: Machine learning
- G16H 20/00: ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method of explaining a machine learning model, including: receiving a plurality of disease states for a patient over time from a database, wherein the disease states have a plurality of features; generating a plurality of locally faithful explanation models for the patient for the disease states based upon the machine learning model; calculating an explanation with respect to one feature of the plurality of features over time using the locally faithful explanation models; and calculating the importance of the one feature of the plurality of features over time based upon the plurality of locally faithful explanation models.

## Claims

### Claim 1

A system, comprising:
a processor; and
a memory coupled with the processor, wherein the memory storing instructions that when executed cause the processor to:
receive a plurality of disease states for a patient over time from a database, wherein the disease states have a plurality of features;
generate a plurality of locally faithful explanation models for the patient for the disease states based upon a machine learning model;
calculate an explanation with respect to one selected feature of the plurality of features over time using the locally faithful explanation models, wherein the locally faithful explanation models include a linear model y=βx, where y is a value of the machine leaning model, x is a feature vector, and β is a vector of explanation coefficients;
calculate an importance of the one selected feature of the plurality of features over time based upon the plurality of locally faithful explanation models; and
provide the importance of the one selected feature of the plurality of features over time for evaluating the plurality of disease states of the patient.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processor is further caused to:
present on a graphical user interface a plot of the explanation and importance of the one feature of the plurality of features over time.

### Claim 3 (depends on 1)

The system of claim 1, wherein the explanation coefficients include explanations for the features in the vector x.

### Claim 4 (depends on 1)

The system of claim 1, wherein the explanation of a feature xk at time t is:
β
k
t
=
Δ
⁢
y
Δ
⁢
x
k
❘
"\[RightBracketingBar]"
t
.

### Claim 5 (depends on 4)

The system of claim 4, wherein the importance of feature xk at time t is:
I
k
t
=
β
k
∑
β
❘
"\[RightBracketingBar]"
t
*
100.

### Claim 6 (depends on 1)

The system of claim 1, wherein generate a plurality of locally faithful explanation models for the patient for the disease states includes optimizing
ξ
⁡
(
x
)
=
min
g
∈
G
L
⁡
(
f
,
g
,
∏
x
)
+
Ω
⁡
(
g
)
,
where ξ(x) is the optimal model explanation that minimizes the sum of loss L and complexity measure, g is a locally faithful explanation model, f is the machine leaning model being explained, L(f, g, Πx) is the measure of how unfaithful g is in approximating f in the locality defined by Πx, and Ω(g) is the complexity of g based upon the number of parameters of g.

### Claim 7 (depends on 6)

The system of claim 6, wherein Ω(g) is the number of non-zero coefficients when g is a linear model.

### Claim 8 (depends on 6)

The system of claim 6, wherein Ω(g) is the depth of a decision three when g is a decision tree.

### Claim 9

A non-transitory machine-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a plurality of disease states for a patient over time from a database, wherein the disease states have a plurality of features;
generating a plurality of locally faithful explanation models for the patient for the disease states based upon the machine learning model;
calculating an explanation with respect to one feature of the plurality of features over time using the locally faithful explanation models, wherein the locally faithful explanation models include a linear model y=βx, where y is a value of the machine leaning model, x is a feature vector, and β is a vector of explanation coefficients;
calculating the average explanation of the one feature of the plurality of features over a specific time period based upon the plurality of locally faithful explanation models; and
providing an importance of the one selected feature of the plurality of features over time for evaluating the plurality of disease states of the patient.

### Claim 10 (depends on 9)

The medium of claim 9, wherein the medium stores further instructions that, when executed by one or more processors, cause the one or more processors to perform presenting on a graphical user interface a plot of the average explanation of the one feature of the plurality of features over the specific time period.

### Claim 11 (depends on 9)

The medium of claim 9, wherein the explanation coefficients include explanations for the features in the vector x.

### Claim 12 (depends on 11)

The medium of claim 11, wherein the explanation of a feature x, at time t is:
β
k
t
=
Δ
⁢
y
Δ
⁢
x
k
❘
"\[RightBracketingBar]"
t
.

### Claim 13 (depends on 12)

The medium of claim 12, wherein the average importance of feature xk over time t1, to t2, is
β
t
1
,
t
2
=
1
t
2
-
t
1
⁢
1
∑
θ
⁢
∑
t
=
t
1
t
2
⁢
θ
t
⁢
β
t
,
where θt is the model confidence.

### Claim 14 (depends on 9)

The medium of claim 9, wherein generating a plurality of locally faithful explanation models for the patient for the disease states includes optimizing
ξ
⁡
(
x
)
=
min
g
∈
G
L
⁡
(
f
,
g
,
∏
x
)
+
Ω
⁡
(
g
)
,
where ξ(x) is the optimal model explanation that minimizes the sum of loss L and complexity measure, g is a locally faithful explanation model, f is the machine leaning model being explained, L(f, g, Πx) is the measure of how unfaithful g is in approximating f in the locality defined by Πx, and Ω(g) is the complexity of g based upon the number of parameters of g.

### Claim 15 (depends on 14)

The medium of claim 14, wherein Ω(g) is the number of non-zero coefficients when g is a linear model.

### Claim 16 (depends on 14)

The medium of claim 14, wherein Ω(g) is the depth of a decision three when g is a decision tree.

### Claim 17

A non-transitory machine-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a plurality of disease states for a patient over time from a database, wherein the disease states have a plurality of features;
generating a plurality of locally faithful explanation models for the patient for the disease states based upon a machine learning model;
calculating an explanation vector based on the plurality of features over time using the locally faithful explanation models, wherein the locally faithful explanation models include a linear model, the linear model having a value of the machine learning model that is based on a feature vector and a vector of explanation coefficients for the feature vector;
training a prediction model to predict the explanation of the machine learning model based upon the explanation vector; and
providing the explanation of the machine learning model over time for evaluating the plurality of disease states of the patient.