# US 2022/0019870 A1 · Verification of classification decisions in convolutional neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0019870 A1
- Document ID: US20220019870A1
- Status: Published application
- Kind: A1
- Application number: 17/294,746
- Filed: 2019-11-12
- Published: 2022-01-20
- Earliest claimed: 2018-11-19
- Assignees: Siemens Aktiengesellschaft
- Applicants: Siemens Aktiengesellschaft
- Inventors: Jindong Gu
- Web page: https://patentgrep.com/patent/US20220019870A1

## CPC

- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

In one aspect the invention relates to a computer-implemented method for providing a computer-implemented method for verifying a visual classification architecture of a convolutional neural network (CNN) and its decisions The method comprises to access (S1) a memory (MEM) with a convolutional neural network (CNN), being trained for a visual classification task into a set of target classes (tc); to use (S2) the convolutional neural network (CNN) for an input image (12) and after a forward pass of the convolutional neural network (CNN), in a backward pass: to apply (S3) a contrastive layer-wise relevance propagation algorithm (CLRP) or to apply (S4) a Bottom Up Attention pattern (BUAP), which is implicitly learned by the convolutional neural network (CNN) for providing (S5) a verification signal (vs).

## Claims

### Claim 1

A computer-implemented method for verifying a visual classification architecture of a convolutional neural network, the method comprising:
accessing a memory with a convolutional neural network, being trained for a visual classification task into a set of target classes; and
using the convolutional neural network for an input image and after a forward pass of the convolutional neural network, in a backward pass:
applying a contrastive layer-wise relevance propagation (CLRP) algorithm or
applying an implicitly learned Bottom Up Attention pattern, to verify a classification ability of the convolutional neural network
for providing a verification signal,
wherein the CLRP algorithm comprises:
generating a first saliency map for each target class of the classification task by a backpropagation algorithm;
calculating a set of virtual classes for each target class, being opposite of the respective target class;
generating a second saliency map for the set of virtual classes by a backpropagation algorithm; and
computing the differences between the first saliency map and the second saliency map for computing a final saliency map.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the verification signal is provided as a saliency map for each feature on each layer of the convolutional neural network.

### Claim 3 (depends on 1)

The method according to claim 1, wherein by applying the CLRP algorithm, class discriminative and instance-specific saliency maps are generated.

### Claim 4 (depends on 2)

The method according to claim 2, wherein for applying an implicitly learned Bottom Up Attention pattern, a deconvolutional CNN algorithm, a gradient backpropagation algorithm or a layer-wise backpropagation algorithm are amended in order to generate saliency maps for features and not for classes.

### Claim 5 (depends on 1)

The method according to claim 1, wherein calculating the virtual class for a specific target class is executed by:
defining any other of the set of target classes as virtual class, or by
defining all other target classes of the set of target classes as virtual class, or by
constructing the virtual class by generating an additional class and connecting it with a last layer using weights, wherein the weights are the inverted weights of the forward pass.

### Claim 6 (depends on 4)

The method according to claim 4, wherein applying the Bottom Up Attention pattern comprises:
collecting and storing all features of the CNN, wherein a feature comprises all activations in a respective layer of the CNN for the input image;
creating a saliency map for each of the features.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the visual classification task is a medical classification task in medical images in order to detect anomalies.

### Claim 8 (depends on 1)

The method according to claim 1, wherein application of the convolutional neural network is only approved, if the provided verification signal is above a pre-configurable confidence threshold.

### Claim 9 (depends on 6)

The method according to claim 6, wherein when applying a Bottom Up Attention pattern for generating a saliency map a guided backpropagation algorithm is used.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the generated saliency maps are post processed and/or may be refined and/or an averaging and/or a thresholding may be applied.

### Claim 11

A verification unit which is configured for verifying a visual classification architecture of a convolutional neural network, comprising:
a memory with the CNN, being trained for a visual classification task into a set of target classes;
a processor which is configured for using the CNN and wherein the processor is configured after a forward pass of the CNN, in a backward pass:
to apply a contrastive layer-wise relevance propagation (CLRP) algorithm or
to apply a Bottom Up Attention pattern, which is implicitly learned by the CNN
for generating a saliency map for each of the target classes,
wherein the CLRP algorithm comprises:
generating a first saliency map for each target class of the classification task by a backpropagation algorithm;
calculating a set of virtual classes for each target class, being opposite of the respective target class;
generating a second saliency map for the set of virtual classes by a backpropagation algorithm; and
computing the differences between the first saliency map and the second saliency map for computing a final saliency map.

### Claim 12 (depends on 1)

A computer program product, comprising a computer readable hardware storage device having computer readable program code stored therein, said program code executable by a processor of a computer system to implement the method according to claim 1, when the program elements are loaded into a memory of the computer.

### Claim 13 (depends on 1)

A computer-readable medium on which a convolutional neural network and program elements are stored that can be read and executed by a computer in order to perform steps of the method according to claim 1, when the program elements are executed by the computer.
