# US 2022/0284435 A1 · System, Method, and Computer Program Product for Determining a Reason for a Deep Learning Model Output

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284435 A1
- Document ID: US20220284435A1
- Status: Published application
- Kind: A1
- Application number: 17/691,402
- Filed: 2022-03-10
- Published: 2022-09-08
- Earliest claimed: 2019-07-18
- Assignees: Visa International Service Association
- Applicants: Visa International Service Association
- Inventors: Hangqi Zhao; Sheng Wang; Dan Wang; Yiwei Cai; Claudia Carolina Barcenas Cardenas
- Web page: https://patentgrep.com/patent/US20220284435A1

## CPC

- G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
- G06Q 20/4016: {involving fraud or risk level assessment in transaction processing}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system, method, and product for determining a reason for a deep learning model output that obtain training data associated with training samples and first labels for the training samples; train a first model using the training samples and the first labels, training the first model generating predictions for the training samples; train a second model using the training samples and the predictions as second labels for the training samples; extract one or more weights of the trained second model; process, using the first model, input data including features associated with at least one sample, to generate output data, the output data including at least one prediction for the at least one sample; and apply the one or more extracted weights to the features to determine one or more contributions of one or more features of the features to the at least one prediction for the at least one sample.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining, with at least one processor, training data associated with a plurality of training samples and a plurality of first labels for the plurality of training samples;
training, with the at least one processor, a first neural network by:
providing, as input to the first neural network, the plurality of training samples,
receiving, as output from the first neural network, a plurality of first predictions for the plurality of training samples, wherein the plurality of first predictions includes a plurality of first probabilities that a plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more first weights of the first neural network according to a first objective function that depends on the plurality of first labels and the plurality of first predictions to generate a first trained neural network;
training, with the at least one processor, a second neural network different than the first neural network by:
providing as input to the second neural network, the plurality of training samples,
receiving, as output from the second neural network, a plurality of second predictions for the plurality of training samples, wherein the plurality of second predictions includes a plurality of second probabilities that the plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more second weights of the second neural network according to a second objective function that depends on the plurality of first predictions as a plurality of second labels for the plurality of training samples and the plurality of second predictions to generate a second trained neural network;
removing, with the at least one processor, from the second trained neural network, the one or more modified second weights;
obtaining, with the at least one processor, transaction data including a plurality of features xi associated with a transaction currently being processed in a transaction processing network;
processing, with the at least one processor, using the trained first neural network, the transaction data including the plurality of features xi associated with the transaction to generate a prediction for the transaction;
determining, by the at least one processor, in real-time during processing of the at least one transaction in the transaction processing network, using the one or more modified second weights removed from the trained second neural network, a contribution score for each feature xi of the plurality of features xi to the prediction for the transaction generated by the first neural network; and
providing, with the at least one processor, to a user device, in real-time during processing of the at least one transaction in the transaction processing network, the prediction and the contribution score for each feature xi as an indication of a contribution of that feature xi to the prediction.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the first neural network includes a deep learning model including a greater number of hidden layers than the second neural network.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
storing, with the at least one processor, in a memory, the one or more modified second weights removed from the trained second neural network.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein processing, using the trained first neural network, the transaction data including the plurality of features xi associated with the transaction includes:
providing, as input to the trained first neural network, the plurality of features xi associated with the transaction,
receiving, as output from the trained first neural network, output data including the prediction for the transaction, wherein the prediction includes a probability that the transaction is a fraudulent transaction; and
authorizing or denying, in the transaction processing network, based on the prediction including the probability that the transaction is a fraudulent transaction, the transaction.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the one or more modified second weights include: weights for hidden units hj in a hidden layer of the trained second neural network, weights for activation units aj in an activation layer of the trained second neural network, weights for output units ym in an output layer of the trained second neural network, a weight matrix uij applied to a feature xi on a path to a hidden unit hj of the trained second neural network, and a weight matrix vjm applied to an output of an activation unit aj on a path to an output unit ym.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the contribution score Contribution (xi->ym) for each feature xi of the plurality of features to the prediction for the transaction received as output from the first neural network is determined according to the following Equations (1) to (4):
h
j
=
∑
i
=
1
K
x
i
*
u
ij
(
1
)
a
j
=
max
⁡
(
0
,
h
j
)
(
2
)
y
m
=
∑
j
=
1
H
a
j
*
v
jm
(
3
)
Contribution
(
x
i
->
y
m
)
=
∑
j
=
1
H
(
x
i
*
u
ij
)
*
(
a
j
*
v
jm
)
(
4
)
wherein Contribution (xi->ym) is the contribution score of the feature xi to the output unit ym, wherein K is a number of the plurality of features xi, wherein H is a number of the hidden units hj in the hidden layer removed from the trained second neural network, wherein M is a number of the output units ym of the output layer removed from the trained second neural network, wherein U and V are weight matrices, wherein a number of weight matrices U is equal to K×H, wherein a number of weight matrices V is equal to H×M, wherein uij is the weight matrix applied to the feature xi on the path to the hidden unit hj removed from the trained second neural network, wherein aj is the activation unit of the activation layer between the hidden layer and the output layer removed from the trained second neural network, and wherein vjm is the weight matrix applied to the output of the activation unit aj on the path to the output unit ym removed from the trained second neural network.

### Claim 7

A computing system comprising:
one or more processors programmed and/or configured to:
obtain training data associated with a plurality of training samples and a plurality of first labels for the plurality of training samples;
train a first neural network by:
providing, as input to the first neural network, the plurality of training samples,
receiving, as output from the first neural network, a plurality of first predictions for the plurality of training samples, wherein the plurality of first predictions includes a plurality of first probabilities that a plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more first weights of the first neural network according to a first objective function that depends on the plurality of first labels and the plurality of first predictions to generate a first trained neural network;
train a second neural network different than the first neural network by:
providing as input to the second neural network, the plurality of training samples,
receiving, as output from the second neural network, a plurality of second predictions for the plurality of training samples, wherein the plurality of second predictions includes a plurality of second probabilities that the plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more second weights of the second neural network according to a second objective function that depends on the plurality of first predictions as a plurality of second labels for the plurality of training samples and the plurality of second predictions to generate a second trained neural network;
remove, from the second trained neural network, the one or more modified second weights;
obtain transaction data including a plurality of features xi associated with a transaction currently being processed in a transaction processing network;
process, using the trained first neural network, the transaction data including the plurality of features xi associated with the transaction to generate a prediction for the transaction;
determine in real-time during processing of the at least one transaction in the transaction processing network, using the one or more modified second weights removed from the trained second neural network, a contribution score for each feature xi of the plurality of features xi to the at least one prediction for the transaction generated by the first neural network; and
provide, with the at least one processor, to a user device, in real-time during processing of the at least one transaction in the transaction processing network, the prediction and the contribution score for each feature xi as an indication of a contribution of that feature xi to the prediction.

### Claim 8 (depends on 7)

The computing system of claim 7, wherein the first neural network includes a deep learning model including a greater number of hidden layers than the second neural network.

### Claim 9 (depends on 7)

The computing system of claim 7, wherein the one or more processors are further programmed and/or configured to:
store, in a memory, the one or more modified second weights removed from the trained second neural network.

### Claim 10 (depends on 7)

The computing system of claim 7, wherein the one or more processors process, using the trained first neural network, the transaction data including the plurality of features xi associated with the transaction by:
providing, as input to the trained first neural network, the plurality of features xi associated with the transaction,
receiving, as output from the trained first neural network, output data including a prediction for the transaction, wherein the prediction includes a probability that the transaction is a fraudulent transaction; and
authorizing or denying, in the transaction processing network, based on the prediction including the probability that the transaction is a fraudulent transaction, the transaction.

### Claim 11 (depends on 10)

The computing system of claim 10, wherein the one or more modified second weights include: weights for hidden units hj in a hidden layer of the trained second neural network, weights for activation units aj in an activation layer of the trained second neural network, weights for output units ym in an output layer of the trained second neural network, a weight matrix uij applied to a feature xi on a path to a hidden unit hj of the trained second neural network, and a weight matrix vjm applied to an output of an activation unit aj on a path to an output unit ym.

### Claim 12 (depends on 11)

The computing system of claim 11, wherein the contribution score for each feature xi of the plurality of features xi to the at least one prediction for the transaction received as output from the first neural network is determined according to the following Equations (1) to (4):
h
j
=
∑
i
=
1
K
x
i
*
u
ij
(
1
)
a
j
=
max
⁡
(
0
,
h
j
)
(
2
)
y
m
=
∑
j
=
1
H
a
j
*
v
jm
(
3
)
Contribution
(
x
i
->
y
m
)
=
∑
j
=
1
H
(
x
i
*
u
ij
)
*
(
a
j
*
v
jm
)
(
4
)
wherein Contribution (xi->ym) is the contribution score of the feature xi to the output unit ym, wherein hj is a number of the plurality of features xi, wherein H is a number of the hidden units hj in the hidden layer removed from the trained second neural network, wherein M is a number of the output units ym of the output layer removed from the trained second neural network, wherein U and V are weight matrices, wherein a number of weight matrices U is equal to K×H, wherein a number of weight matrices V is equal to H×M, wherein uij is the weight matrix applied to the feature xi on the path to the hidden unit hj removed from the trained second neural network, wherein aj is the activation unit of the activation layer between the hidden layer and the output layer removed from the trained second neural network, and wherein vjm is the weight matrix applied to the output of the activation unit aj on the path to the output unit ym removed from the trained second neural network.

### Claim 13

A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:
obtain training data associated with a plurality of training samples and a plurality of first labels for the plurality of training samples;
train a first neural network by:
providing, as input to the first neural network, the plurality of training samples,
receiving, as output from the first neural network, a plurality of first predictions for the plurality of training samples, wherein the plurality of first predictions includes a plurality of first probabilities that a plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more first weights of the first neural network according to a first objective function that depends on the plurality of first labels and the plurality of first predictions to generate a first trained neural network;
train a second neural network different than the first neural network by:
providing as input to the second neural network, the plurality of training samples,
receiving, as output from the second neural network, a plurality of second predictions for the plurality of training samples, wherein the plurality of second predictions includes a plurality of second probabilities that the plurality of transactions associated with the plurality of training samples are fraudulent transactions, and
modifying one or more second weights of the second neural network according to a second objective function that depends on the plurality of first predictions as a plurality of second labels for the plurality of training samples and the plurality of second predictions to generate a second trained neural network,
remove, from the second trained neural network, the one or more modified second weights;
obtain transaction data including a plurality of features xi associated with a transaction currently being processed in a transaction processing network;
process, using the trained first neural network, the transaction data including the plurality of features xi associated with the transaction to generate a prediction for the transaction;
determine in real-time during processing of the at least one transaction in the transaction processing network, using the one or more modified second weights removed from the trained second neural network, a contribution score for each feature xi of the plurality of features xi to the at least one prediction for the transaction generated by the first neural network; and
provide, with the at least one processor, to a user device, in real-time during processing of the at least one transaction in the transaction processing network, the prediction and the contribution score Contribution (xi->ym) for each feature xi as an indication of a contribution of that feature xi to the prediction.

### Claim 14 (depends on 13)

The computer program product of claim 13, wherein the first neural network includes a deep learning model including a greater number of hidden layers than the second neural network.

### Claim 15 (depends on 13)

The computer program product of claim 13, wherein the program instructions, when executed by at least one processor, further cause the at least one processor to:
store, in a memory, the one or more modified second weights removed from the trained second neural network.

### Claim 16 (depends on 13)

The computer program product of claim 13, wherein the program instructions, when executed by at least one processor, further cause the at least one processor to process, using the trained first neural network, the transaction data including the plurality of features xi associated with the transaction by:
providing, as input to the trained first neural network, the plurality of features xi associated with the transaction,
receiving, as output from the trained first neural network, output data including a prediction for the transaction, wherein the prediction includes a probability that the transaction is a fraudulent transaction; and
authorizing or denying, in the transaction processing network, based on the prediction including the probability that the transaction is a fraudulent transaction, the transaction.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein the one or more modified second weights include: weights for hidden units hj in a hidden layer of the trained second neural network, weights for activation units aj in an activation layer of the trained second neural network, weights for output units ym in an output layer of the trained second neural network, a weight matrix uij applied to a feature xi on a path to a hidden unit hj of the trained second neural network, and a weight matrix vjm applied to an output of an activation unit aj on a path to an output unit ym.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein the contribution score Contribution (xi->ym) for each feature xi of the plurality of features xi to the prediction for the transaction received as output from the first neural network is determined according to the following Equations (1) to (4):
h
j
=
∑
i
=
1
K
x
i
*
u
ij
(
1
)
a
j
=
max
⁡
(
0
,
h
j
)
(
2
)
y
m
=
∑
j
=
1
H
a
j
*
v
jm
(
3
)
Contribution
(
x
i
->
y
m
)
=
∑
j
=
1
H
(
x
i
*
u
ij
)
*
(
a
j
*
v
jm
)
(
4
)
wherein Contribution (xi->ym) is the contribution score of the feature xi to the output unit ym, wherein K is a number of the plurality of features xi, wherein H is a number of the hidden units hj in the hidden layer removed from the trained second neural network, wherein M is a number of the output units ym of the output layer removed from the trained second neural network, wherein U and V are weight matrices, wherein a number of weight matrices U is equal to K×H, wherein a number of weight matrices V is equal to H×M, wherein uij is the weight matrix applied to the feature xi on the path to the hidden unit hj removed from the trained second neural network, wherein aj is the activation unit of the activation layer between the hidden layer and the output layer removed from the trained second neural network, and wherein vjm is the weight matrix applied to the output of the activation unit aj on the path to the output unit ym removed from the trained second neural network.
