# US 12,182,698 B2 · Interpretable visualization system for graph neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,698 B2
- Document ID: US12182698B2
- Status: Granted
- Kind: B2
- Application number: 17/039,989
- Filed: 2020-09-30
- Published: 2022-03-31
- Granted: 2024-12-31
- Earliest claimed: 2020-09-30
- Assignees: International Business Machines Corporation; Massachusetts Institute of Technology
- Applicants: International Business Machines Corporation; Massachusetts Institute of Technology
- Inventors: Dakuo Wang; Sijia Liu; Abel Valente; Chuang Gan; Bei Chen; Dongyu Liu; Yi Sun
- Web page: https://patentgrep.com/patent/US12182698B2

## CPC

- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Use a computerized trained graph neural network model to classify an input instance with a predicted label. With a computerized graph neural network interpretation module, compute a gradient-based saliency matrix based on the input instance and the predicted label, by taking a partial derivative of class prediction with respect to an adjacency matrix of the model. With a computerized user interface, obtain user input responsive to the gradient-based saliency matrix. Optionally, modify the trained graph neural network model based on the user input; and re-classify the input instance with a new predicted label based on the modified trained graph neural network model.

## Claims

### Claim 1

A method comprising:
using a computerized trained graph neural network model to classify an input instance with a predicted label;
with a computerized graph neural network interpretation module, computing a gradient-based saliency matrix based on said input instance and said predicted label, by taking a partial derivative of class prediction with respect to an adjacency matrix of said model;
rendering said gradient-based saliency matrix in a computerized user interface;
with said computerized user interface, obtaining user input responsive to said gradient-based saliency matrix, wherein said user input is responsive to said rendering; and
determining a minimum perturbation to add to said adjacency matrix that maximizes prediction accuracy of said model, wherein:
said rendering showing information related to said model; and
said user input comprises at least one of:
adding a missing edge based on a gradient-based saliency map-based graph explanation to improve said prediction accuracy; and
at least one of derating and removing an unimportant edge based on the gradient-based saliency map-based graph explanation to improve said prediction accuracy.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
modifying said trained graph neural network model based on said user input; and
re-classifying said input instance with a new predicted label based on said modified trained graph neural network model.

### Claim 3 (depends on 1)

The method of claim 1, wherein
said user input comprises at least one of:
adding another missing edge to boost adversarial attack performance; and
removing another existing edge to boost said adversarial attack performance.

### Claim 4 (depends on 1)

The method of claim 1, wherein said input instance comprises an image to be recognized.

### Claim 5 (depends on 1)

The method of claim 1, wherein said rendering includes at least one of a global network, an instance sub-network, an instance full adjacent matrix, and an instance applied adjacent matrix with a threshold applied.

### Claim 6

An apparatus comprising:
a memory; and
at least one processor, coupled to said memory, and operative to:
instantiate a computerized trained graph neural network model, a computerized graph neural network interpretation module, and a computerized user interface;
use said computerized trained graph neural network model to classify an input instance with a predicted label;
with said computerized graph neural network interpretation module, compute a gradient-based saliency matrix based on said input instance and said predicted label, by taking a partial derivative of class prediction with respect to an adjacency matrix of said model;
rendering said gradient-based saliency matrix in the computerized user interface;
with said computerized user interface, obtain user input responsive to said gradient-based saliency matrix, wherein said user input is responsive to said rendering; and
determining a minimum perturbation to add to said adjacency matrix that maximizes prediction accuracy of said model, wherein:
said rendering showing information related to said model; and
said user input comprises at least one of:
adding a missing edge based on a gradient-based saliency map-based graph explanation to improve said prediction accuracy; and
at least one of derating and removing an unimportant edge based on the gradient-based saliency map-based graph explanation to improve said prediction accuracy.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein said at least one processor is further operative to:
modify said trained graph neural network model based on said user input; and
re-classify said input instance with a new predicted label based on said modified trained graph neural network model.

### Claim 8 (depends on 6)

The apparatus of claim 6, wherein said user input comprises at least one of:
adding another missing edge to boost adversarial attack performance; and
removing another existing edge to boost said adversarial attack performance.

### Claim 9 (depends on 6)

The apparatus of claim 6, wherein said input instance comprises an image to be recognized.

### Claim 10 (depends on 6)

The apparatus of claim 6, wherein said computerized user interface is configured to render at least one of a global network, an instance sub-network, an instance full adjacent matrix, and an instance applied adjacent matrix with a threshold applied.

### Claim 11

A non-transitory computer readable medium comprising computer executable instructions which when executed by a computer cause the computer to perform a method of:
instantiating a computerized trained graph neural network model, a computerized graph neural network interpretation module, and a computerized user interface;
using said computerized trained graph neural network model to classify an input instance with a predicted label;
with said computerized graph neural network interpretation module, computing a gradient-based saliency matrix based on said input instance and said predicted label, by taking a partial derivative of class prediction with respect to an adjacency matrix of said model;
rendering said gradient-based saliency matrix in the computerized user interface;
with said computerized user interface, obtaining user input responsive to said gradient-based saliency matrix, wherein said user input is responsive to said rendering; and
determining a minimum perturbation to add to said adjacency matrix that maximizes prediction accuracy of said model, wherein:
said rendering showing information related to said model; and
said user input comprises at least one of:
adding a missing edge based on a gradient-based saliency map-based graph explanation to improve said prediction accuracy; and
at least one of derating and removing an unimportant edge based on the gradient-based saliency map-based graph explanation to improve said prediction accuracy.

### Claim 12 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the performed method further comprises:
modifying said trained graph neural network model based on said user input; and
re-classifying said input instance with a new predicted label based on said modified trained graph neural network model.

### Claim 13 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein said user input comprises at least one of:
adding another missing edge to boost adversarial attack performance; and
removing another existing edge to boost said adversarial attack performance.

### Claim 14 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein said input instance comprises an image to be recognized.